2023-11-17T05:21:26.696 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-17T05:21:26.696 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284 --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} --name yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/orig.config.yaml 2023-11-17T05:21:26.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T05:21:26.815 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284 branch: wip-yuri3-testing-2023-11-14-1227-pacific description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} email: null first_in_suite: false job_id: '7461284' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-11-14-1227-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac workunit: branch: wip-yuri3-testing-2023-11-14-1227-pacific sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 7739 sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-11-14-1227-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE= tasks: - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip: null - cephadm.shell: host.a: - ceph orch device ls --refresh - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - ceph nfs cluster create foo --ingress --virtual-ip {{VIP0}}/{{VIPPREFIXLEN}} --port 2999 - ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - cephadm.wait_for_service: service: nfs.foo - cephadm.wait_for_service: service: ingress.nfs.foo - vip.exec: host.a: - mkdir /mnt/foo - sleep 5 - mount -t nfs {{VIP0}}:/fake /mnt/foo -o port=2999 - echo test > /mnt/foo/testfile - sync - cephadm.shell: host.a: - "echo \"Check with each haproxy down in turn...\"\nfor haproxy in `ceph orch\ \ ps | grep ^haproxy.nfs.foo. | awk '{print $1}'`; do\n ceph orch daemon stop\ \ $haproxy\n while ! ceph orch ps | grep $haproxy | grep stopped; do sleep\ \ 1 ; done\n cat /mnt/foo/testfile\n echo $haproxy > /mnt/foo/testfile\n \ \ sync\n ceph orch daemon start $haproxy\n while ! ceph orch ps | grep $haproxy\ \ | grep running; do sleep 1 ; done\ndone\n" volumes: - /mnt/foo:/mnt/foo - vip.exec: all-hosts: - "echo \"Check with $(hostname) ganesha(s) down...\"\nfor c in `systemctl | grep\ \ ceph- | grep @nfs | awk '{print $1}'`; do\n cid=`echo $c | sed 's/@/-/'`\n\ \ id=`echo $c | cut -d @ -f 2 | sed 's/.service$//'`\n fsid=`echo $c |\ \ cut -d @ -f 1 | cut -d - -f 2-`\n echo \"Removing daemon $id fsid $fsid...\"\ \n sudo $TESTDIR/cephadm rm-daemon --fsid $fsid --name $id\n\n echo \"\ Waking up cephadm...\"\n sudo $TESTDIR/cephadm shell -- ceph orch ps --refresh\n\ \n while ! timeout 1 cat /mnt/foo/testfile ; do true ; done\n echo \"\ Mount is back!\"\ndone\n" - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_22:29:26 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-17T05:21:26.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa; will attempt to use it 2023-11-17T05:21:26.816 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks 2023-11-17T05:21:26.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-17T05:21:26.817 INFO:teuthology.task.internal:Checking packages... 2023-11-17T05:21:26.841 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac' 2023-11-17T05:21:26.841 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-17T05:21:26.841 INFO:teuthology.packaging:ref: None 2023-11-17T05:21:26.841 INFO:teuthology.packaging:tag: None 2023-11-17T05:21:26.842 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T05:21:26.842 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:21:26.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T05:21:27.006 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-171-ged3c8e92-1bionic 2023-11-17T05:21:27.006 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-17T05:21:27.013 INFO:teuthology.task.internal:no buildpackages task found 2023-11-17T05:21:27.013 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-17T05:21:27.072 INFO:teuthology.task.internal:Saving configuration 2023-11-17T05:21:27.087 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-17T05:21:27.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T05:21:27.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 05:14:41.215224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4U6ItCfOdoN1WdROC3KO40lcw/kzXeIM8M8q950kfL'} 2023-11-17T05:21:27.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 05:14:41.217500', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4U6ItCfOdoN1WdROC3KO40lcw/kzXeIM8M8q950kfL'} 2023-11-17T05:21:27.171 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-17T05:21:27.220 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['host.a', 'client.0'] 2023-11-17T05:21:27.220 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['host.b', 'client.1'] 2023-11-17T05:21:27.221 INFO:teuthology.run_tasks:Running task console_log... 2023-11-17T05:21:27.323 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb9222930d0>, signals=[15]) 2023-11-17T05:21:27.323 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-17T05:21:27.330 INFO:teuthology.task.internal:Opening connections... 2023-11-17T05:21:27.330 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T05:21:27.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:21:27.407 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-11-17T05:21:27.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:21:27.476 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-17T05:21:27.483 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2023-11-17T05:21:27.535 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2023-11-17T05:21:27.535 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-11-17T05:21:27.588 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2023-11-17T05:21:27.588 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="18.04" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-17T05:21:27.589 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-17T05:21:27.590 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=bionic 2023-11-17T05:21:27.590 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=bionic 2023-11-17T05:21:27.591 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-11-17T05:21:27.617 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-11-17T05:21:27.622 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-11-17T05:21:27.623 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Ubuntu" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:ID=ubuntu 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE=debian 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="18.04" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-17T05:21:27.671 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_CODENAME=bionic 2023-11-17T05:21:27.672 INFO:teuthology.orchestra.run.smithi156.stdout:UBUNTU_CODENAME=bionic 2023-11-17T05:21:27.672 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-11-17T05:21:27.692 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-17T05:21:27.701 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-17T05:21:27.775 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-17T05:21:27.775 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2023-11-17T05:21:27.777 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-11-17T05:21:27.785 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-17T05:21:27.808 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-17T05:21:27.808 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2023-11-17T05:21:27.833 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-11-17T05:21:27.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-17T05:21:27.912 INFO:teuthology.run_tasks:Running task kernel... 2023-11-17T05:21:27.929 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-17T05:21:27.929 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-17T05:21:27.930 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T05:21:27.930 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T05:21:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:21:27.935 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-11-17T05:21:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:21:27.937 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-11-17T05:21:27.982 INFO:teuthology.orchestra.run.smithi156.stdout:4.15.0-200-generic 2023-11-17T05:21:27.983 INFO:teuthology.task.kernel:Running kernel on smithi156: 4.15.0-200-generic 2023-11-17T05:21:27.983 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get clean 2023-11-17T05:21:27.984 INFO:teuthology.orchestra.run.smithi006.stdout:4.15.0-200-generic 2023-11-17T05:21:27.984 INFO:teuthology.task.kernel:Running kernel on smithi006: 4.15.0-200-generic 2023-11-17T05:21:27.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2023-11-17T05:21:28.079 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get update 2023-11-17T05:21:28.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2023-11-17T05:21:28.234 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2023-11-17T05:21:28.234 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2023-11-17T05:21:28.254 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2023-11-17T05:21:28.254 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2023-11-17T05:21:28.301 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 2023-11-17T05:21:28.321 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 2023-11-17T05:21:28.365 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2023-11-17T05:21:28.366 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2023-11-17T05:21:28.592 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,666 kB] 2023-11-17T05:21:28.607 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,666 kB] 2023-11-17T05:21:28.693 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,045 kB] 2023-11-17T05:21:28.714 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,045 kB] 2023-11-17T05:21:28.734 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [554 kB] 2023-11-17T05:21:28.739 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [39.7 kB] 2023-11-17T05:21:28.739 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,347 kB] 2023-11-17T05:21:28.749 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [554 kB] 2023-11-17T05:21:28.754 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [187 kB] 2023-11-17T05:21:28.755 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,347 kB] 2023-11-17T05:21:28.756 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,915 kB] 2023-11-17T05:21:28.768 INFO:teuthology.orchestra.run.smithi156.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [39.7 kB] 2023-11-17T05:21:28.769 INFO:teuthology.orchestra.run.smithi156.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [187 kB] 2023-11-17T05:21:28.772 INFO:teuthology.orchestra.run.smithi156.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,663 kB] 2023-11-17T05:21:28.777 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,663 kB] 2023-11-17T05:21:28.788 INFO:teuthology.orchestra.run.smithi156.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,915 kB] 2023-11-17T05:21:28.794 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [421 kB] 2023-11-17T05:21:28.798 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [25.6 kB] 2023-11-17T05:21:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.2 kB] 2023-11-17T05:21:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,088 B] 2023-11-17T05:21:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [53.3 kB] 2023-11-17T05:21:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [53.2 kB] 2023-11-17T05:21:28.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [18.1 kB] 2023-11-17T05:21:28.800 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [18.2 kB] 2023-11-17T05:21:28.807 INFO:teuthology.orchestra.run.smithi156.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [421 kB] 2023-11-17T05:21:28.811 INFO:teuthology.orchestra.run.smithi156.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.2 kB] 2023-11-17T05:21:28.811 INFO:teuthology.orchestra.run.smithi156.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [25.6 kB] 2023-11-17T05:21:28.812 INFO:teuthology.orchestra.run.smithi156.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,088 B] 2023-11-17T05:21:28.812 INFO:teuthology.orchestra.run.smithi156.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [53.3 kB] 2023-11-17T05:21:28.815 INFO:teuthology.orchestra.run.smithi156.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [53.2 kB] 2023-11-17T05:21:28.815 INFO:teuthology.orchestra.run.smithi156.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [18.1 kB] 2023-11-17T05:21:28.815 INFO:teuthology.orchestra.run.smithi156.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [18.2 kB] 2023-11-17T05:21:28.899 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,379 kB] 2023-11-17T05:21:28.909 INFO:teuthology.orchestra.run.smithi156.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,379 kB] 2023-11-17T05:21:29.350 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,717 kB] 2023-11-17T05:21:29.358 INFO:teuthology.orchestra.run.smithi156.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,717 kB] 2023-11-17T05:21:29.486 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [467 kB] 2023-11-17T05:21:29.493 INFO:teuthology.orchestra.run.smithi156.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [467 kB] 2023-11-17T05:21:29.502 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,317 kB] 2023-11-17T05:21:29.510 INFO:teuthology.orchestra.run.smithi156.stdout:Get:24 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [33.0 kB] 2023-11-17T05:21:29.511 INFO:teuthology.orchestra.run.smithi156.stdout:Get:25 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,317 kB] 2023-11-17T05:21:29.534 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [33.0 kB] 2023-11-17T05:21:29.535 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [182 kB] 2023-11-17T05:21:29.538 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,303 kB] 2023-11-17T05:21:29.542 INFO:teuthology.orchestra.run.smithi156.stdout:Get:26 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [182 kB] 2023-11-17T05:21:29.546 INFO:teuthology.orchestra.run.smithi156.stdout:Get:27 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,078 kB] 2023-11-17T05:21:29.582 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,078 kB] 2023-11-17T05:21:29.587 INFO:teuthology.orchestra.run.smithi156.stdout:Get:28 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,303 kB] 2023-11-17T05:21:29.613 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [308 kB] 2023-11-17T05:21:29.622 INFO:teuthology.orchestra.run.smithi156.stdout:Get:29 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [308 kB] 2023-11-17T05:21:29.622 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,008 B] 2023-11-17T05:21:29.628 INFO:teuthology.orchestra.run.smithi156.stdout:Get:30 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.8 kB] 2023-11-17T05:21:29.783 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.8 kB] 2023-11-17T05:21:29.792 INFO:teuthology.orchestra.run.smithi156.stdout:Get:31 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,008 B] 2023-11-17T05:21:29.875 INFO:teuthology.orchestra.run.smithi156.stdout:Get:32 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [3,928 B] 2023-11-17T05:21:29.954 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [3,928 B] 2023-11-17T05:21:31.644 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 20.1 MB in 3s (6,095 kB/s) 2023-11-17T05:21:31.771 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 20.1 MB in 3s (5,879 kB/s) 2023-11-17T05:21:33.426 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2023-11-17T05:21:33.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-17T05:21:33.522 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2023-11-17T05:21:33.570 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2023-11-17T05:21:33.573 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-17T05:21:33.654 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2023-11-17T05:21:33.862 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2023-11-17T05:21:33.862 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2023-11-17T05:21:33.921 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2023-11-17T05:21:33.921 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2023-11-17T05:21:34.090 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2023-11-17T05:21:34.090 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2023-11-17T05:21:34.090 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-4.15.0-213-generic 2023-11-17T05:21:34.091 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-4.15.0-213-generic linux-modules-extra-4.15.0-213-generic 2023-11-17T05:21:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2023-11-17T05:21:34.093 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2023-11-17T05:21:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2023-11-17T05:21:34.093 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2023-11-17T05:21:34.141 INFO:teuthology.orchestra.run.smithi156.stdout:The following additional packages will be installed: 2023-11-17T05:21:34.142 INFO:teuthology.orchestra.run.smithi156.stdout: linux-generic linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2023-11-17T05:21:34.142 INFO:teuthology.orchestra.run.smithi156.stdout: linux-headers-generic linux-image-4.15.0-213-generic 2023-11-17T05:21:34.142 INFO:teuthology.orchestra.run.smithi156.stdout: linux-modules-4.15.0-213-generic linux-modules-extra-4.15.0-213-generic 2023-11-17T05:21:34.144 INFO:teuthology.orchestra.run.smithi156.stdout:Suggested packages: 2023-11-17T05:21:34.145 INFO:teuthology.orchestra.run.smithi156.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2023-11-17T05:21:34.145 INFO:teuthology.orchestra.run.smithi156.stdout:Recommended packages: 2023-11-17T05:21:34.145 INFO:teuthology.orchestra.run.smithi156.stdout: thermald 2023-11-17T05:21:34.157 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2023-11-17T05:21:34.158 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2023-11-17T05:21:34.158 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-4.15.0-213-generic linux-modules-4.15.0-213-generic 2023-11-17T05:21:34.158 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-4.15.0-213-generic 2023-11-17T05:21:34.160 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2023-11-17T05:21:34.160 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-17T05:21:34.211 INFO:teuthology.orchestra.run.smithi156.stdout:The following NEW packages will be installed: 2023-11-17T05:21:34.211 INFO:teuthology.orchestra.run.smithi156.stdout: linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2023-11-17T05:21:34.211 INFO:teuthology.orchestra.run.smithi156.stdout: linux-image-4.15.0-213-generic linux-modules-4.15.0-213-generic 2023-11-17T05:21:34.211 INFO:teuthology.orchestra.run.smithi156.stdout: linux-modules-extra-4.15.0-213-generic 2023-11-17T05:21:34.213 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages will be upgraded: 2023-11-17T05:21:34.214 INFO:teuthology.orchestra.run.smithi156.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-17T05:21:34.346 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 164 not upgraded. 2023-11-17T05:21:34.346 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 67.4 MB of archives. 2023-11-17T05:21:34.346 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 351 MB of additional disk space will be used. 2023-11-17T05:21:34.347 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-213-generic amd64 4.15.0-213.224 [13.4 MB] 2023-11-17T05:21:34.399 INFO:teuthology.orchestra.run.smithi156.stdout:3 upgraded, 5 newly installed, 0 to remove and 164 not upgraded. 2023-11-17T05:21:34.399 INFO:teuthology.orchestra.run.smithi156.stdout:Need to get 67.4 MB of archives. 2023-11-17T05:21:34.400 INFO:teuthology.orchestra.run.smithi156.stdout:After this operation, 351 MB of additional disk space will be used. 2023-11-17T05:21:34.400 INFO:teuthology.orchestra.run.smithi156.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-213-generic amd64 4.15.0-213.224 [13.4 MB] 2023-11-17T05:21:35.390 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-213-generic amd64 4.15.0-213.224 [8,111 kB] 2023-11-17T05:21:35.443 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-213-generic amd64 4.15.0-213.224 [8,111 kB] 2023-11-17T05:21:35.638 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-213-generic amd64 4.15.0-213.224 [33.7 MB] 2023-11-17T05:21:35.650 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-213-generic amd64 4.15.0-213.224 [33.7 MB] 2023-11-17T05:21:36.654 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.213.196 [1,864 B] 2023-11-17T05:21:36.655 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.213.196 [2,460 B] 2023-11-17T05:21:36.655 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213 all 4.15.0-213.224 [11.0 MB] 2023-11-17T05:21:36.750 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.213.196 [1,864 B] 2023-11-17T05:21:36.750 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.213.196 [2,460 B] 2023-11-17T05:21:36.750 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213 all 4.15.0-213.224 [11.0 MB] 2023-11-17T05:21:36.964 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213-generic amd64 4.15.0-213.224 [1,250 kB] 2023-11-17T05:21:37.004 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.213.196 [2,360 B] 2023-11-17T05:21:37.083 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213-generic amd64 4.15.0-213.224 [1,250 kB] 2023-11-17T05:21:37.118 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.213.196 [2,360 B] 2023-11-17T05:21:37.697 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 67.4 MB in 3s (23.9 MB/s) 2023-11-17T05:21:37.822 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 67.4 MB in 3s (23.4 MB/s) 2023-11-17T05:21:37.870 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-4.15.0-213-generic. 2023-11-17T05:21:38.011 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-modules-4.15.0-213-generic. 2023-11-17T05:21:38.793 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 135963 files and directories currently installed.) 2023-11-17T05:21:38.797 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:38.866 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:38.966 INFO:teuthology.orchestra.run.smithi156.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 135963 files and directories currently installed.) 2023-11-17T05:21:38.970 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../0-linux-modules-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:39.048 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:40.509 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-4.15.0-213-generic. 2023-11-17T05:21:40.522 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:40.623 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:40.757 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-image-4.15.0-213-generic. 2023-11-17T05:21:40.771 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../1-linux-image-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:40.862 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:41.137 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-213-generic. 2023-11-17T05:21:41.152 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:41.196 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:41.401 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-213-generic. 2023-11-17T05:21:41.416 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:41.460 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:45.040 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:45.166 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:45.328 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../3-linux-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:45.486 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:45.492 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:45.620 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:45.813 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-4.15.0-213. 2023-11-17T05:21:45.930 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-4.15.0-213_4.15.0-213.224_all.deb ... 2023-11-17T05:21:45.965 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-image-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:45.973 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-4.15.0-213 (4.15.0-213.224) ... 2023-11-17T05:21:46.251 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-headers-4.15.0-213. 2023-11-17T05:21:46.268 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../5-linux-headers-4.15.0-213_4.15.0-213.224_all.deb ... 2023-11-17T05:21:46.336 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-4.15.0-213 (4.15.0-213.224) ... 2023-11-17T05:21:50.835 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-4.15.0-213-generic. 2023-11-17T05:21:50.859 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:50.893 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:51.205 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-headers-4.15.0-213-generic. 2023-11-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../6-linux-headers-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2023-11-17T05:21:51.271 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:53.353 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:53.482 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:53.712 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.213.196_amd64.deb ... 2023-11-17T05:21:53.783 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:53.857 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2023-11-17T05:21:54.193 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:54.419 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:54.820 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:54.923 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:21:54.953 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-200-generic 2023-11-17T05:21:54.953 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:21:54.953 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-213-generic 2023-11-17T05:21:55.054 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:55.316 INFO:teuthology.orchestra.run.smithi156.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:21:55.316 INFO:teuthology.orchestra.run.smithi156.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-200-generic 2023-11-17T05:21:55.316 INFO:teuthology.orchestra.run.smithi156.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:21:55.316 INFO:teuthology.orchestra.run.smithi156.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-213-generic 2023-11-17T05:21:55.456 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:55.639 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-4.15.0-213 (4.15.0-213.224) ... 2023-11-17T05:21:55.882 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (4.15.0.213.196) ... 2023-11-17T05:21:55.991 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:55.999 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-4.15.0-213 (4.15.0-213.224) ... 2023-11-17T05:21:56.117 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (4.15.0.213.196) ... 2023-11-17T05:21:56.143 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-image-generic (4.15.0.213.196) ... 2023-11-17T05:21:56.243 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (4.15.0.213.196) ... 2023-11-17T05:21:56.268 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:56.394 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-generic (4.15.0.213.196) ... 2023-11-17T05:21:56.394 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:56.520 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-generic (4.15.0.213.196) ... 2023-11-17T05:21:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-17T05:21:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:21:56.663 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2023-11-17T05:21:56.856 INFO:teuthology.orchestra.run.smithi156.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-17T05:21:56.856 INFO:teuthology.orchestra.run.smithi156.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:22:12.822 INFO:teuthology.orchestra.run.smithi156.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-17T05:22:31.330 INFO:teuthology.orchestra.run.smithi156.stdout:Sourcing file `/etc/default/grub' 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Generating grub configuration file ... 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2023-11-17T05:22:31.401 INFO:teuthology.orchestra.run.smithi156.stdout:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2023-11-17T05:22:31.402 INFO:teuthology.orchestra.run.smithi156.stdout:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2023-11-17T05:22:31.402 INFO:teuthology.orchestra.run.smithi156.stdout:done 2023-11-17T05:22:31.402 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-17T05:22:31.402 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2023-11-17T05:22:31.436 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2023-11-17T05:22:31.436 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2023-11-17T05:22:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:done 2023-11-17T05:22:31.440 DEBUG:teuthology.orchestra.run.smithi156:> dpkg -s linux-image-generic 2023-11-17T05:22:31.441 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2023-11-17T05:22:31.463 INFO:teuthology.orchestra.run.smithi156.stdout:Package: linux-image-generic 2023-11-17T05:22:31.463 INFO:teuthology.orchestra.run.smithi156.stdout:Status: install ok installed 2023-11-17T05:22:31.463 INFO:teuthology.orchestra.run.smithi156.stdout:Priority: optional 2023-11-17T05:22:31.463 INFO:teuthology.orchestra.run.smithi156.stdout:Section: kernel 2023-11-17T05:22:31.463 INFO:teuthology.orchestra.run.smithi156.stdout:Installed-Size: 18 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: amd64 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Source: linux-meta 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Version: 4.15.0.213.196 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Recommends: thermald 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout:Description: Generic Linux kernel image 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout: This package will always depend on the latest generic kernel image 2023-11-17T05:22:31.464 INFO:teuthology.orchestra.run.smithi156.stdout: available. 2023-11-17T05:22:31.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-200-generic Expected: 4.15.0-213-generic 2023-11-17T05:22:31.464 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T05:22:31.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T05:22:31.465 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-17T05:22:31.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T05:22:31.465 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get clean 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 18 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 4.15.0.213.196 2023-11-17T05:22:31.466 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2023-11-17T05:22:31.467 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T05:22:31.467 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2023-11-17T05:22:31.467 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2023-11-17T05:22:31.467 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2023-11-17T05:22:31.467 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2023-11-17T05:22:31.467 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-200-generic Expected: 4.15.0-213-generic 2023-11-17T05:22:31.467 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T05:22:31.467 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T05:22:31.467 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-17T05:22:31.467 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T05:22:31.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2023-11-17T05:22:31.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2023-11-17T05:22:31.581 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get update 2023-11-17T05:22:31.714 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2023-11-17T05:22:31.720 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2023-11-17T05:22:31.737 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2023-11-17T05:22:31.834 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2023-11-17T05:22:31.845 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2023-11-17T05:22:31.845 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2023-11-17T05:22:31.915 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2023-11-17T05:22:31.996 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2023-11-17T05:22:33.887 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2023-11-17T05:22:33.932 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2023-11-17T05:22:33.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-17T05:22:33.982 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-17T05:22:34.042 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2023-11-17T05:22:34.123 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2023-11-17T05:22:34.297 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2023-11-17T05:22:34.298 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2023-11-17T05:22:34.397 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2023-11-17T05:22:34.397 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2023-11-17T05:22:34.520 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (4.15.0.213.196). 2023-11-17T05:22:34.520 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2023-11-17T05:22:34.520 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2023-11-17T05:22:34.520 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2023-11-17T05:22:34.521 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-4.15.0-194-generic 2023-11-17T05:22:34.521 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-17T05:22:34.575 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 164 not upgraded. 2023-11-17T05:22:34.577 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2023-11-17T05:22:34.597 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2023-11-17T05:22:34.597 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 18 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 4.15.0.213.196 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2023-11-17T05:22:34.598 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2023-11-17T05:22:34.599 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-11-17T05:22:34.604 INFO:teuthology.orchestra.run.smithi156.stdout:linux-image-generic is already the newest version (4.15.0.213.196). 2023-11-17T05:22:34.604 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages were automatically installed and are no longer required: 2023-11-17T05:22:34.604 INFO:teuthology.orchestra.run.smithi156.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2023-11-17T05:22:34.604 INFO:teuthology.orchestra.run.smithi156.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2023-11-17T05:22:34.604 INFO:teuthology.orchestra.run.smithi156.stdout: linux-modules-extra-4.15.0-194-generic 2023-11-17T05:22:34.605 INFO:teuthology.orchestra.run.smithi156.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-17T05:22:34.644 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.sSXxILKn8O 2023-11-17T05:22:34.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.sSXxILKn8O 2023-11-17T05:22:34.661 INFO:teuthology.orchestra.run.smithi156.stdout:0 upgraded, 0 newly installed, 0 to remove and 164 not upgraded. 2023-11-17T05:22:34.663 DEBUG:teuthology.orchestra.run.smithi156:> dpkg -s linux-image-generic 2023-11-17T05:22:34.682 INFO:teuthology.orchestra.run.smithi156.stdout:Package: linux-image-generic 2023-11-17T05:22:34.682 INFO:teuthology.orchestra.run.smithi156.stdout:Status: install ok installed 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Priority: optional 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Section: kernel 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Installed-Size: 18 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: amd64 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Source: linux-meta 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Version: 4.15.0.213.196 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Recommends: thermald 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout:Description: Generic Linux kernel image 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout: This package will always depend on the latest generic kernel image 2023-11-17T05:22:34.683 INFO:teuthology.orchestra.run.smithi156.stdout: available. 2023-11-17T05:22:34.684 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-11-17T05:22:34.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.sSXxILKn8O 2023-11-17T05:22:34.735 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.zQLfxdLcYv 2023-11-17T05:22:34.735 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zQLfxdLcYv 2023-11-17T05:22:34.793 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.zQLfxdLcYv 2023-11-17T05:22:34.836 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.sSXxILKn8O is 10KB 2023-11-17T05:22:34.848 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.sSXxILKn8O 2023-11-17T05:22:34.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-17T05:22:34.914 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:22:34.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-17T05:22:34.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-17T05:22:34.930 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.zQLfxdLcYv is 10KB 2023-11-17T05:22:34.942 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.zQLfxdLcYv 2023-11-17T05:22:34.950 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-17T05:22:34.987 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-213-generic 2023-11-17T05:22:34.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2023-11-17T05:22:35.015 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:22:35.016 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-17T05:22:35.016 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-17T05:22:35.105 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-213-generic 2023-11-17T05:22:35.106 DEBUG:teuthology.orchestra.run.smithi156:> sudo update-grub 2023-11-17T05:22:36.502 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2023-11-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2023-11-17T05:22:36.580 INFO:teuthology.orchestra.run.smithi156.stderr:Sourcing file `/etc/default/grub' 2023-11-17T05:22:36.598 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2023-11-17T05:22:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:22:36.824 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:22:36.873 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2023-11-17T05:22:36.885 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2023-11-17T05:22:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:22:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2023-11-17T05:22:37.226 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2023-11-17T05:22:37.238 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2023-11-17T05:22:37.260 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2023-11-17T05:22:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2023-11-17T05:22:37.314 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2023-11-17T05:22:37.327 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2023-11-17T05:22:37.855 INFO:teuthology.orchestra.run.smithi006.stderr:done 2023-11-17T05:22:37.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2023-11-17T05:22:38.534 INFO:teuthology.orchestra.run.smithi156.stderr:done 2023-11-17T05:22:38.535 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2023-11-17T05:23:07.889 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-17T05:23:13.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:23:13.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:23:13.916 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-17T05:23:13.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:23:13.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:23:32.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-17T05:23:34.631 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-11-17T05:23:43.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:24:00.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:24:01.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:24:01.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:24:04.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-17T05:24:16.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:24:16.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:24:18.107 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-17T05:24:20.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-11-17T05:24:32.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:24:32.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:24:32.734 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-11-17T05:24:33.059 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:24:33.059 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.15.0-213-generic"... 2023-11-17T05:24:33.060 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-11-17T05:24:33.109 INFO:teuthology.orchestra.run.smithi006.stdout:4.15.0-213-generic 2023-11-17T05:24:33.109 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-213-generic vs 4.15.0-213-generic 2023-11-17T05:24:33.109 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T05:24:33.109 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-17T05:24:34.110 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-17T05:24:34.110 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T05:24:34.142 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2023-11-17T05:24:34.170 DEBUG:teuthology.parallel:result is None 2023-11-17T05:24:35.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:24:35.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:24:35.724 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-11-17T05:24:36.015 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:24:36.016 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.15.0-213-generic"... 2023-11-17T05:24:36.016 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-11-17T05:24:36.062 INFO:teuthology.orchestra.run.smithi156.stdout:4.15.0-213-generic 2023-11-17T05:24:36.062 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-213-generic vs 4.15.0-213-generic 2023-11-17T05:24:36.062 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T05:24:36.062 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-17T05:24:37.063 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-17T05:24:41.449 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T05:24:41.530 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2023-11-17T05:24:41.551 DEBUG:teuthology.parallel:result is None 2023-11-17T05:24:41.551 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-17T05:24:41.559 INFO:teuthology.task.internal:Creating test directory... 2023-11-17T05:24:41.559 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T05:24:41.561 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T05:24:41.567 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-17T05:24:41.618 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-17T05:24:41.666 INFO:teuthology.task.internal:Creating archive directory... 2023-11-17T05:24:41.667 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T05:24:41.680 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T05:24:41.705 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-17T05:24:41.722 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-17T05:24:41.722 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T05:24:41.734 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T05:24:41.750 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:24:41.765 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:24:41.769 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:24:41.777 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:24:41.779 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-17T05:24:41.785 INFO:teuthology.task.internal:Configuring sudo... 2023-11-17T05:24:41.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T05:24:41.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T05:24:41.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-17T05:24:41.852 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-17T05:24:41.852 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T05:24:41.869 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T05:24:41.887 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:24:41.921 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:24:41.973 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:24:41.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T05:24:42.036 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:24:42.044 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:24:42.091 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:24:42.091 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T05:24:42.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2023-11-17T05:24:42.160 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-11-17T05:24:42.251 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-17T05:24:42.259 INFO:teuthology.task.internal:Starting timer... 2023-11-17T05:24:42.259 INFO:teuthology.run_tasks:Running task pcp... 2023-11-17T05:24:42.297 INFO:teuthology.run_tasks:Running task selinux... 2023-11-17T05:24:42.331 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2023-11-17T05:24:42.331 DEBUG:teuthology.task.selinux:Excluding smithi156: OS 'ubuntu' does not support SELinux 2023-11-17T05:24:42.332 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-17T05:24:42.332 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-17T05:24:42.332 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-17T05:24:42.332 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-17T05:24:42.362 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-17T05:24:42.362 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-17T05:24:42.385 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-17T05:24:42.388 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-17T05:30:48.598 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2023-11-17T05:30:48.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:30:48.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:30:48.681 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-11-17T05:30:48.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-11-17T05:30:48.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:30:48.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:30:48.873 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-11-17T05:30:48.992 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-17T05:30:48.993 INFO:teuthology.run_tasks:Running task clock... 2023-11-17T05:30:49.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-17T05:30:49.007 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T05:30:49.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:30:49.010 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T05:30:49.010 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:30:49.075 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: ntpd 4.2.8p10@1.3728-o (1): Starting 2023-11-17T05:30:49.075 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Command line: ntpd -gq 2023-11-17T05:30:49.076 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: proto: precision = 0.127 usec (-23) 2023-11-17T05:30:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T05:30:49.077 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T05:30:49.077 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-17T05:30:49.079 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: restrict ::: KOD does nothing without LIMITED. 2023-11-17T05:30:49.079 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen and drop on 0 v6wildcard [::]:123 2023-11-17T05:30:49.079 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-17T05:30:49.079 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen normally on 2 lo 127.0.0.1:123 2023-11-17T05:30:49.080 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen normally on 3 ens1f0 172.21.15.6:123 2023-11-17T05:30:49.080 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen normally on 4 lo [::1]:123 2023-11-17T05:30:49.080 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%3]:123 2023-11-17T05:30:49.080 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:49 ntpd[24186]: Listening on routing socket on fd #22 for interface updates 2023-11-17T05:30:49.091 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: ntpd 4.2.8p10@1.3728-o (1): Starting 2023-11-17T05:30:49.091 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Command line: ntpd -gq 2023-11-17T05:30:49.091 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: proto: precision = 0.123 usec (-23) 2023-11-17T05:30:49.092 INFO:teuthology.orchestra.run.smithi156.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T05:30:49.092 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T05:30:49.092 INFO:teuthology.orchestra.run.smithi156.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-17T05:30:49.093 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: restrict ::: KOD does nothing without LIMITED. 2023-11-17T05:30:49.093 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen and drop on 0 v6wildcard [::]:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen normally on 2 lo 127.0.0.1:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen normally on 3 enp3s0f1 172.21.15.156:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen normally on 4 lo [::1]:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6dfd%5]:123 2023-11-17T05:30:49.094 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:49 ntpd[24181]: Listening on routing socket on fd #22 for interface updates 2023-11-17T05:30:50.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:50 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:50.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:50 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:50.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:50 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:50.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:50 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:50.091 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:50.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:50 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:52 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:52 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:52 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:52 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:52.099 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:52.099 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:52.100 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:52.100 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:52.100 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:52.100 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:52.142 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:52.142 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:52 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:53.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:53 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:53.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:53 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:53.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:53 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:53.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:53 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:54.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:54.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:54.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:54.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:54.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:54.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:54.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:54.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:54 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:54.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:54 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:55 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:55 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:55 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:55 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:56.076 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:56.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:56.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:56.077 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:56.077 INFO:teuthology.orchestra.run.smithi006.stdout:17 Nov 05:30:56 ntpd[24186]: ntpd: time slew +0.000678 s 2023-11-17T05:30:56.078 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew +0.000678s 2023-11-17T05:30:56.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-17T05:30:56.078 INFO:teuthology.orchestra.run.smithi006.stderr:17 Nov 05:30:56 ntpd[24186]: can't open /var/log/ntpstats/loopstats.20231117: Permission denied 2023-11-17T05:30:56.091 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T05:30:56.091 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T05:30:56.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T05:30:56.092 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T05:30:56.092 INFO:teuthology.orchestra.run.smithi156.stdout:17 Nov 05:30:56 ntpd[24181]: ntpd: time slew +0.000360 s 2023-11-17T05:30:56.092 INFO:teuthology.orchestra.run.smithi156.stdout:ntpd: time slew +0.000360s 2023-11-17T05:30:56.093 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-17T05:30:56.093 INFO:teuthology.orchestra.run.smithi156.stderr:17 Nov 05:30:56 ntpd[24181]: can't open /var/log/ntpstats/loopstats.20231117: Permission denied 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.139 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.162 INFO:teuthology.orchestra.run.smithi156.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T05:30:56.162 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================== 2023-11-17T05:30:56.162 INFO:teuthology.orchestra.run.smithi156.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.162 INFO:teuthology.orchestra.run.smithi156.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.163 INFO:teuthology.orchestra.run.smithi156.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.163 INFO:teuthology.orchestra.run.smithi156.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T05:30:56.163 INFO:teuthology.run_tasks:Running task nvme_loop... 2023-11-17T05:30:56.182 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2023-11-17T05:30:56.183 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:30:56.184 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:30:56.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-17T05:30:56.193 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 350 Links: 1 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:29:20.367618057 +0000 2023-11-17T05:30:56.247 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:29:20.047620550 +0000 2023-11-17T05:30:56.248 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:29:20.047620550 +0000 2023-11-17T05:30:56.248 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:30:56.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-17T05:30:56.301 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:30:56.302 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:30:56.302 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000397847 s, 1.3 MB/s 2023-11-17T05:30:56.303 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-17T05:30:56.353 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 357 Links: 1 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:29:20.511616937 +0000 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:29:20.507616968 +0000 2023-11-17T05:30:56.403 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:29:20.507616968 +0000 2023-11-17T05:30:56.404 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:30:56.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-17T05:30:56.457 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:30:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:30:56.458 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000368687 s, 1.4 MB/s 2023-11-17T05:30:56.459 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-17T05:30:56.507 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2023-11-17T05:30:56.558 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-17T05:30:56.558 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 363 Links: 1 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:29:21.143611880 +0000 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:29:20.903613885 +0000 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:29:20.903613885 +0000 2023-11-17T05:30:56.559 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:30:56.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-17T05:30:56.613 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:30:56.613 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:30:56.613 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000328759 s, 1.6 MB/s 2023-11-17T05:30:56.614 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-17T05:30:56.663 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2023-11-17T05:30:56.714 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 369 Links: 1 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:29:21.535608438 +0000 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:29:21.275610721 +0000 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:29:21.275610721 +0000 2023-11-17T05:30:56.715 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:30:56.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-17T05:30:56.769 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:30:56.769 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:30:56.769 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000315109 s, 1.6 MB/s 2023-11-17T05:30:56.770 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-17T05:30:56.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-11-17T05:30:56.928 INFO:teuthology.orchestra.run.smithi006.stdout:loop 2023-11-17T05:30:56.931 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_1... 2023-11-17T05:30:56.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-11-17T05:30:56.954 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-11-17T05:30:56.982 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_11 2023-11-17T05:30:57.001 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_2... 2023-11-17T05:30:57.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-11-17T05:30:57.067 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-11-17T05:30:57.095 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_21 2023-11-17T05:30:57.116 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_3... 2023-11-17T05:30:57.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-11-17T05:30:57.179 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-11-17T05:30:57.202 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_31 2023-11-17T05:30:57.231 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_4... 2023-11-17T05:30:57.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-11-17T05:30:57.297 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-11-17T05:30:57.322 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_41 2023-11-17T05:30:57.343 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:30:57.343 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:30:57.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme list 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:Node SN Model Namespace Usage Format FW Rev 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme0n1 CVFT5330000H400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme1n1 b9ec99303a4fe1b1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme2n1 f7ff5a34bd09e3d4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme3n1 ac7d093a0a4c141b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:57.458 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme4n1 55e1634ade73856b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:57.459 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T05:30:57.459 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:30:57.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/scratch_devs 2023-11-17T05:30:57.519 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:30:57.519 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:30:57.526 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-17T05:30:57.527 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 338 Links: 1 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:29:29.844861308 +0000 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:29:29.620863278 +0000 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:29:29.620863278 +0000 2023-11-17T05:30:57.578 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:30:57.578 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-17T05:30:57.632 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:30:57.632 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:30:57.632 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000280898 s, 1.8 MB/s 2023-11-17T05:30:57.633 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-17T05:30:57.684 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 345 Links: 1 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:29:30.216858036 +0000 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:29:30.004859901 +0000 2023-11-17T05:30:57.735 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:29:30.004859901 +0000 2023-11-17T05:30:57.736 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:30:57.736 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-17T05:30:57.790 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:30:57.790 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:30:57.791 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.0002863 s, 1.8 MB/s 2023-11-17T05:30:57.792 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-17T05:30:57.844 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-11-17T05:30:57.899 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-17T05:30:57.899 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:57.899 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 351 Links: 1 2023-11-17T05:30:57.899 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:57.899 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:29:30.564854975 +0000 2023-11-17T05:30:57.900 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:29:30.348856875 +0000 2023-11-17T05:30:57.900 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:29:30.348856875 +0000 2023-11-17T05:30:57.900 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:30:57.900 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-17T05:30:57.953 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:30:57.954 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:30:57.954 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000279951 s, 1.8 MB/s 2023-11-17T05:30:57.955 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-17T05:30:58.004 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 357 Links: 1 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:29:30.920851844 +0000 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:29:30.692853849 +0000 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:29:30.692853849 +0000 2023-11-17T05:30:58.055 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:30:58.056 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-17T05:30:58.110 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:30:58.110 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:30:58.110 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000280734 s, 1.8 MB/s 2023-11-17T05:30:58.111 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-17T05:30:58.160 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-11-17T05:30:58.279 INFO:teuthology.orchestra.run.smithi156.stdout:loop 2023-11-17T05:30:58.280 INFO:tasks.nvme_loop:Connecting nvme_loop smithi156:/dev/vg_nvme/lv_1... 2023-11-17T05:30:58.281 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-11-17T05:30:58.298 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-17T05:30:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/vg_nvme/lv_11 2023-11-17T05:30:58.348 INFO:tasks.nvme_loop:Connecting nvme_loop smithi156:/dev/vg_nvme/lv_2... 2023-11-17T05:30:58.348 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-11-17T05:30:58.416 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-17T05:30:58.444 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/vg_nvme/lv_21 2023-11-17T05:30:58.463 INFO:tasks.nvme_loop:Connecting nvme_loop smithi156:/dev/vg_nvme/lv_3... 2023-11-17T05:30:58.464 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-11-17T05:30:58.529 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-17T05:30:58.553 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/vg_nvme/lv_31 2023-11-17T05:30:58.575 INFO:tasks.nvme_loop:Connecting nvme_loop smithi156:/dev/vg_nvme/lv_4... 2023-11-17T05:30:58.575 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-11-17T05:30:58.642 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-17T05:30:58.669 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/vg_nvme/lv_41 2023-11-17T05:30:58.690 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:30:58.690 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:30:58.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo nvme list 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:Node SN Model Namespace Usage Format FW Rev 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/nvme0n1 PHMB74760040480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/nvme1n1 52956b7a9abd9528 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/nvme2n1 55a86bf49532c310 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/nvme3n1 76b4d1da7df5bd44 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/nvme4n1 bd01f4b66a9c929e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.15.0-2 2023-11-17T05:30:58.798 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T05:30:58.798 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:30:58.798 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/scratch_devs 2023-11-17T05:30:58.856 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-17T05:30:58.974 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-17T05:30:58.974 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:30:58.975 INFO:tasks.cephadm:Cluster fsid is 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:30:58.975 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-17T05:30:58.975 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-17T05:30:58.975 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi006': '172.21.15.6', 'mon.smithi156': '172.21.15.156'} 2023-11-17T05:30:58.975 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-17T05:30:58.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname $(hostname -s) 2023-11-17T05:30:58.995 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2023-11-17T05:30:59.011 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref ed3c8e926b9de93026c8a7c1d86be5d6821b52ac)... 2023-11-17T05:30:59.012 DEBUG:teuthology.orchestra.run.smithi006:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.274 INFO:teuthology.orchestra.run.smithi006.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Nov 17 05:30 /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.274 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.524 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Nov 17 05:30 /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.524 DEBUG:teuthology.orchestra.run.smithi006:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.533 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T05:30:59.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2023-11-17T05:30:59.592 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2023-11-17T05:30:59.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /etc/ceph 2023-11-17T05:30:59.650 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2023-11-17T05:30:59.666 INFO:tasks.cephadm:Writing seed config... 2023-11-17T05:30:59.667 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-17T05:30:59.668 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-17T05:30:59.668 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-17T05:30:59.668 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-17T05:30:59.668 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-17T05:30:59.668 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-17T05:30:59.669 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-17T05:30:59.669 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-17T05:30:59.671 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:30:59.671 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-17T05:30:59.704 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7d14d910-850a-11ee-8dbe-212e2dc638e7 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-11-17T05:30:59.704 DEBUG:teuthology.orchestra.run.smithi006:mon.smithi006> sudo journalctl -f -n 0 -u ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006.service 2023-11-17T05:30:59.749 INFO:tasks.cephadm:Bootstrapping... 2023-11-17T05:30:59.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac -v bootstrap --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:31:00.124 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2023-11-17T05:31:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:-------------------------------------------------------------------------------- 2023-11-17T05:31:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', '-v', 'bootstrap', '--fsid', '7d14d910-850a-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.6', '--skip-admin-label'] 2023-11-17T05:31:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying podman|docker is present... 2023-11-17T05:31:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying lvm2 is present... 2023-11-17T05:31:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying time synchronization is in place... 2023-11-17T05:31:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T05:31:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-11-17T05:31:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: enabled 2023-11-17T05:31:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.279 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-11-17T05:31:00.287 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.296 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: enabled 2023-11-17T05:31:00.306 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: active 2023-11-17T05:31:00.307 INFO:teuthology.orchestra.run.smithi006.stderr:Unit ntp.service is enabled and running 2023-11-17T05:31:00.307 INFO:teuthology.orchestra.run.smithi006.stderr:Repeating the final host check... 2023-11-17T05:31:00.307 INFO:teuthology.orchestra.run.smithi006.stderr:docker (/usr/bin/docker) is present 2023-11-17T05:31:00.307 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl is present 2023-11-17T05:31:00.308 INFO:teuthology.orchestra.run.smithi006.stderr:lvcreate is present 2023-11-17T05:31:00.315 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T05:31:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-11-17T05:31:00.341 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: enabled 2023-11-17T05:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.368 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-11-17T05:31:00.376 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: inactive 2023-11-17T05:31:00.384 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: enabled 2023-11-17T05:31:00.392 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: active 2023-11-17T05:31:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:Unit ntp.service is enabled and running 2023-11-17T05:31:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:Host looks OK 2023-11-17T05:31:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:Cluster fsid: 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:Acquiring lock 139833894380824 on /run/cephadm/7d14d910-850a-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:Lock 139833894380824 acquired on /run/cephadm/7d14d910-850a-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying IP 172.21.15.6 port 3300 ... 2023-11-17T05:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying IP 172.21.15.6 port 6789 ... 2023-11-17T05:31:00.394 INFO:teuthology.orchestra.run.smithi006.stderr:Base mon IP(s) is [172.21.15.6:3300, 172.21.15.6:6789], mon addrv is [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2023-11-17T05:31:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 2023-11-17T05:31:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T05:31:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 2023-11-17T05:31:00.403 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-17T05:31:00.404 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1684sec hoplimit 64 pref medium 2023-11-17T05:31:00.407 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-17T05:31:00.407 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-17T05:31:00.408 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T05:31:00.408 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: 3: ens1f0: mtu 1500 state UP qlen 1000 2023-11-17T05:31:00.408 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15e4/64 scope link 2023-11-17T05:31:00.408 INFO:teuthology.orchestra.run.smithi006.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T05:31:00.408 INFO:teuthology.orchestra.run.smithi006.stderr:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2023-11-17T05:31:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2023-11-17T05:31:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-17T05:31:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-17T05:31:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac... 2023-11-17T05:31:00.763 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac: Pulling from ceph-ci/ceph 2023-11-17T05:31:00.765 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 837e66e3ff89: Pulling fs layer 2023-11-17T05:31:00.766 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 0b35c5e1bfec: Pulling fs layer 2023-11-17T05:31:01.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 837e66e3ff89: Verifying Checksum 2023-11-17T05:31:01.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 837e66e3ff89: Download complete 2023-11-17T05:31:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 0b35c5e1bfec: Verifying Checksum 2023-11-17T05:31:03.318 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 0b35c5e1bfec: Download complete 2023-11-17T05:31:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 837e66e3ff89: Pull complete 2023-11-17T05:31:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: 0b35c5e1bfec: Pull complete 2023-11-17T05:31:14.669 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: Digest: sha256:f94e7380824ed7d68c996c4c1e29ccce9634d15592a525871e481b74ad1cb87c 2023-11-17T05:31:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: Status: Downloaded newer image for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:31:14.727 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/docker: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:31:24.496 INFO:teuthology.orchestra.run.smithi006.stderr:ceph: ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable) 2023-11-17T05:31:24.993 INFO:teuthology.orchestra.run.smithi006.stderr:Ceph version: ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable) 2023-11-17T05:31:24.993 INFO:teuthology.orchestra.run.smithi006.stderr:Extracting ceph user uid/gid from container image... 2023-11-17T05:31:25.861 INFO:teuthology.orchestra.run.smithi006.stderr:stat: 167 167 2023-11-17T05:31:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:Creating initial keys... 2023-11-17T05:31:27.210 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-authtool: AQCv+lZlVQtrDBAAebFbuk96wI0lM5IQUHGnYw== 2023-11-17T05:31:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-authtool: AQCw+lZlloZLHhAAFYL+GYCOA+KmmbyUq0Qv3A== 2023-11-17T05:31:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-authtool: AQCx+lZlcLIcMBAAfzuXN3CMIRmimXyuiVHWiw== 2023-11-17T05:31:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:Creating initial monmap... 2023-11-17T05:31:31.144 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T05:31:31.144 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:31.145 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T05:31:31.597 INFO:teuthology.orchestra.run.smithi006.stderr:monmaptool for smithi006 [v2:172.21.15.6:3300,v1:172.21.15.6:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T05:31:31.597 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/monmaptool: set fsid to 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:31.597 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T05:31:31.597 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:31:31.597 INFO:teuthology.orchestra.run.smithi006.stderr:Creating mon... 2023-11-17T05:31:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T05:31:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 1 imported monmap: 2023-11-17T05:31:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-17T05:31:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: last_changed 2023-11-17T05:31:31.143165+0000 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: created 2023-11-17T05:31:31.143165+0000 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.smithi006 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 0 /usr/bin/ceph-mon: set fsid to 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Compile date Nov 14 2023 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: DB SUMMARY 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi006/store.db dir, Total Num: 0, files: 2023-11-17T05:31:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi006/store.db: 2023-11-17T05:31:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.error_if_exists: 0 2023-11-17T05:31:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.create_if_missing: 1 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.env: 0x55bf7b0f8080 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.fs: Posix File System 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.info_log: 0x55bf7d5313a0 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.statistics: (nil) 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.use_fsync: 0 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T05:31:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.db_log_dir: 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi006/store.db 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T05:31:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.write_buffer_manager: 0x55bf7d537980 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T05:31:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.unordered_write: 0 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.row_cache: None 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.wal_filter: None 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T05:31:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.two_write_queues: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.atomic_flush: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T05:31:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T05:31:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T05:31:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T05:31:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T05:31:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T05:31:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.max_open_files: -1 2023-11-17T05:31:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Compression algorithms supported: 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kZSTD supported: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kXpressCompression supported: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kZlibCompression supported: 1 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.478+0000 7f1e29be9880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T05:31:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000001 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.merge_operator: 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_filter: None 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bf7d445d38) 2023-11-17T05:31:32.517 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_cache: 0x55bf7d47cd10 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-17T05:31:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-17T05:31:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression: NoCompression 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T05:31:32.520 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.num_levels: 7 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T05:31:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T05:31:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T05:31:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.table_properties_collectors: 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T05:31:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.bloom_locality: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.ttl: 2592000 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.482+0000 7f1e29be9880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.486+0000 7f1e29be9880 4 rocksdb: DB pointer 0x55bf7d545800 2023-11-17T05:31:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.486+0000 7f1e12a70700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.486+0000 7f1e12a70700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:32.526 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:31:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: 2023-11-17T05:31:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.490+0000 7f1e29be9880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-17T05:31:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.490+0000 7f1e29be9880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-17T05:31:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:31:32.490+0000 7f1e29be9880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi006 for mon.smithi006 2023-11-17T05:31:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:create mon.smithi006 on 2023-11-17T05:31:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2023-11-17T05:31:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target -> /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target. 2023-11-17T05:31:33.214 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target -> /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target. 2023-11-17T05:31:33.471 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to reset failed state of unit ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006.service: Unit ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006.service is not loaded. 2023-11-17T05:31:33.478 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target.wants/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006.service -> /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@.service. 2023-11-17T05:31:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:firewalld does not appear to be present 2023-11-17T05:31:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T05:31:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mon to start... 2023-11-17T05:31:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mon... 2023-11-17T05:31:33.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:33 smithi006 systemd[1]: Started Ceph mon.smithi006 for 7d14d910-850a-11ee-8dbe-212e2dc638e7. 2023-11-17T05:31:35.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:35 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.035677+0000 mon.smithi006 (mon.0) 0 : [INF] mkfs 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:35.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:35 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.021564+0000 mon.smithi006 (mon.0) 1 : cluster [INF] mon.smithi006 is new leader, mons smithi006 in quorum (ranks 0) 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: cluster: 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: id: 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: services: 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi006 (age 0.29087s) 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-17T05:31:35.325 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: data: 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: pgs: 2023-11-17T05:31:35.326 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:35.814 INFO:teuthology.orchestra.run.smithi006.stderr:mon is available 2023-11-17T05:31:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:Assimilating anything we can from ceph.conf... 2023-11-17T05:31:36.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.030800+0000 mon.smithi006 (mon.0) 2 : cluster [INF] mon.smithi006 is new leader, mons smithi006 in quorum (ranks 0) 2023-11-17T05:31:36.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.032680+0000 mon.smithi006 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0]} removed_ranks: {} 2023-11-17T05:31:36.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.039335+0000 mon.smithi006 (mon.0) 4 : cluster [DBG] fsmap 2023-11-17T05:31:36.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.048225+0000 mon.smithi006 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T05:31:36.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: cluster 2023-11-17T05:31:35.049690+0000 mon.smithi006 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T05:31:36.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:36 smithi006 bash[25367]: audit 2023-11-17T05:31:35.321594+0000 mon.smithi006 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.6:0/128017096' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-17T05:31:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [global] 2023-11-17T05:31:36.969 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: fsid = 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:36.969 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2023-11-17T05:31:36.969 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-17T05:31:36.969 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [mgr] 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [osd] 2023-11-17T05:31:36.971 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-17T05:31:36.971 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-17T05:31:37.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:37 smithi006 bash[25367]: audit 2023-11-17T05:31:36.962666+0000 mon.smithi006 (mon.0) 8 : audit [INF] from='client.? 172.21.15.6:0/1479433082' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T05:31:37.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:37 smithi006 bash[25367]: audit 2023-11-17T05:31:36.966301+0000 mon.smithi006 (mon.0) 9 : audit [INF] from='client.? 172.21.15.6:0/1479433082' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-17T05:31:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:Generating new minimal ceph.conf... 2023-11-17T05:31:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:Restarting the monitor... 2023-11-17T05:31:39.056 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:38 smithi006 bash[25367]: audit 2023-11-17T05:31:38.616184+0000 mon.smithi006 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.6:0/112166318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:31:39.343 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 systemd[1]: Stopping Ceph mon.smithi006 for 7d14d910-850a-11ee-8dbe-212e2dc638e7... 2023-11-17T05:31:39.343 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 bash[25897]: Error response from daemon: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-mon.smithi006 2023-11-17T05:31:39.343 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 bash[25367]: debug 2023-11-17T05:31:39.222+0000 7f14b368f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi006 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T05:31:39.343 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 bash[25367]: debug 2023-11-17T05:31:39.222+0000 7f14b368f700 -1 mon.smithi006@0(leader) e1 *** Got Signal Terminated *** 2023-11-17T05:31:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2023-11-17T05:31:40.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 bash[25897]: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-mon-smithi006 2023-11-17T05:31:40.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 systemd[1]: Stopped Ceph mon.smithi006 for 7d14d910-850a-11ee-8dbe-212e2dc638e7. 2023-11-17T05:31:40.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:39 smithi006 systemd[1]: Started Ceph mon.smithi006 for 7d14d910-850a-11ee-8dbe-212e2dc638e7. 2023-11-17T05:31:41.132 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.050+0000 7fe995cf7880 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.050+0000 7fe995cf7880 0 ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable), process ceph-mon, pid 7 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.050+0000 7fe995cf7880 0 pidfile_write: ignore empty --pid-file 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 0 load: jerasure load: lrc load: isa 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Compile date Nov 14 2023 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: DB SUMMARY 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: CURRENT file: CURRENT 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi006/store.db dir, Total Num: 1, files: 000004.sst 2023-11-17T05:31:41.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi006/store.db: 000006.log size: 86479 ; 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.error_if_exists: 0 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.create_if_missing: 0 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.env: 0x55a6ee6b4080 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.fs: Posix File System 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.info_log: 0x55a6efe58580 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.statistics: (nil) 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.use_fsync: 0 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T05:31:41.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T05:31:41.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.db_log_dir: 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi006/store.db 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T05:31:41.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.write_buffer_manager: 0x55a6f0aa6480 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T05:31:41.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T05:31:41.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T05:31:41.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.unordered_write: 0 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.row_cache: None 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.wal_filter: None 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T05:31:41.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.two_write_queues: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.atomic_flush: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T05:31:41.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_open_files: -1 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Compression algorithms supported: 2023-11-17T05:31:41.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kZSTD supported: 0 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kXpressCompression supported: 0 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kZlibCompression supported: 1 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000005 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.merge_operator: 2023-11-17T05:31:41.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_filter: None 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a6efdfbd30) 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cache_index_and_filter_blocks: 1 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: pin_top_level_index_and_filter: 1 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: index_type: 0 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: data_block_index_type: 0 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: index_shortening: 1 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: data_block_hash_table_util_ratio: 0.750000 2023-11-17T05:31:41.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: hash_index_allow_collision: 1 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: checksum: 1 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: no_block_cache: 0 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_cache: 0x55a6efe32f10 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_cache_name: BinnedLRUCache 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_cache_options: 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: capacity : 536870912 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: num_shard_bits : 4 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: strict_capacity_limit : 0 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: high_pri_pool_ratio: 0.000 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_cache_compressed: (nil) 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: persistent_cache: (nil) 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_size: 4096 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_size_deviation: 10 2023-11-17T05:31:41.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_restart_interval: 16 2023-11-17T05:31:41.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: index_block_restart_interval: 1 2023-11-17T05:31:41.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: metadata_block_size: 4096 2023-11-17T05:31:41.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: partition_filters: 0 2023-11-17T05:31:41.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: use_delta_encoding: 1 2023-11-17T05:31:41.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: whole_key_filtering: 1 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: verify_compression: 0 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: read_amp_bytes_per_bit: 0 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: format_version: 2 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: enable_index_compression: 1 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: block_align: 0 2023-11-17T05:31:41.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression: NoCompression 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.num_levels: 7 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T05:31:41.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T05:31:41.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T05:31:41.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T05:31:41.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T05:31:41.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T05:31:41.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T05:31:41.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T05:31:41.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T05:31:41.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T05:31:41.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T05:31:41.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T05:31:41.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T05:31:41.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.table_properties_collectors: 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T05:31:41.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.bloom_locality: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.ttl: 2592000 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-17T05:31:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199101063020, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.058+0000 7fe995cf7880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe995cf7880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe995cf7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199101064322, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83352, "table_properties": {"data_size": 81619, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76131, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700199101, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe995cf7880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe995cf7880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199101065246, "job": 1, "event": "recovery_finished"} 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe995cf7880 4 rocksdb: DB pointer 0x55a6efefb800 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe97cafd700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.062+0000 7fe97cafd700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** DB Stats ** 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:31:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** Compaction Stats [default] ** 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: L0 2/0 83.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Sum 2/0 83.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** Compaction Stats [default] ** 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:41.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:41.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:41.187 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Cumulative compaction: 0.00 GB write, 15.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Interval compaction: 0.00 GB write, 15.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** Compaction Stats [default] ** 2023-11-17T05:31:41.188 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: L0 2/0 83.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Sum 2/0 83.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** Compaction Stats [default] ** 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:31:41.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Cumulative compaction: 0.00 GB write, 15.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 0 starting mon.smithi006 rank 0 at public addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] at bind addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi006 fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 1 mon.smithi006@-1(???) e1 preinit fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 0 mon.smithi006@-1(???).mds e1 new map 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 0 mon.smithi006@-1(???).mds e1 print_map 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: e1 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: legacy client fscid: -1 2023-11-17T05:31:41.190 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: 2023-11-17T05:31:41.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: No filesystems configured 2023-11-17T05:31:41.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 0 mon.smithi006@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-17T05:31:41.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 0 mon.smithi006@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:31:41.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: audit 2023-11-17T05:31:38.616184+0000 mon.smithi006 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.6:0/112166318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:31:41.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: debug 2023-11-17T05:31:41.070+0000 7fe995cf7880 1 mon.smithi006@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-17T05:31:41.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cluster 2023-11-17T05:31:41.078278+0000 mon.smithi006 (mon.0) 1 : cluster [INF] mon.smithi006 is new leader, mons smithi006 in quorum (ranks 0) 2023-11-17T05:31:41.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cluster 2023-11-17T05:31:41.078360+0000 mon.smithi006 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0]} removed_ranks: {} 2023-11-17T05:31:41.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cluster 2023-11-17T05:31:41.078834+0000 mon.smithi006 (mon.0) 3 : cluster [DBG] fsmap 2023-11-17T05:31:41.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cluster 2023-11-17T05:31:41.078882+0000 mon.smithi006 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T05:31:41.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:41 smithi006 bash[26022]: cluster 2023-11-17T05:31:41.079332+0000 mon.smithi006 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T05:31:41.906 INFO:teuthology.orchestra.run.smithi006.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-17T05:31:41.906 INFO:teuthology.orchestra.run.smithi006.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:31:41.906 INFO:teuthology.orchestra.run.smithi006.stderr:Creating mgr... 2023-11-17T05:31:41.907 INFO:teuthology.orchestra.run.smithi006.stderr:Verifying port 9283 ... 2023-11-17T05:31:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Failed to reset failed state of unit ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mgr.smithi006.odexej.service: Unit ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mgr.smithi006.odexej.service is not loaded. 2023-11-17T05:31:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7.target.wants/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mgr.smithi006.odexej.service -> /etc/systemd/system/ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@.service. 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:firewalld does not appear to be present 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:firewalld does not appear to be present 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mgr to start... 2023-11-17T05:31:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mgr... 2023-11-17T05:31:42.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:42 smithi006 bash[26022]: audit 2023-11-17T05:31:41.373101+0000 mon.smithi006 (mon.0) 6 : audit [INF] from='client.? 172.21.15.6:0/3097569086' entity='client.admin' 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsid": "7d14d910-850a-11ee-8dbe-212e2dc638e7", 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:31:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 0 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:31:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "smithi006" 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:31:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:31:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:31:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:31:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "available": false, 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:31:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "nfs", 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:31:35.037081+0000", 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:31:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:31:44.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:44 smithi006 bash[26022]: audit 2023-11-17T05:31:44.031313+0000 mon.smithi006 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.6:0/2710664975' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:31:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:mgr not available, waiting (1/15)... 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsid": "7d14d910-850a-11ee-8dbe-212e2dc638e7", 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:31:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 0 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "smithi006" 2023-11-17T05:31:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_age": 6, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:31:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:31:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:31:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:31:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:31:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "available": false, 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "nfs", 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:31:35.037081+0000", 2023-11-17T05:31:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:31:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:31:48.144 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:47 smithi006 bash[26022]: audit 2023-11-17T05:31:47.747798+0000 mon.smithi006 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.6:0/1318627530' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:31:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:mgr not available, waiting (2/15)... 2023-11-17T05:31:50.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: cluster 2023-11-17T05:31:50.341138+0000 mon.smithi006 (mon.0) 9 : cluster [INF] Activating manager daemon smithi006.odexej 2023-11-17T05:31:59.572 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: cluster 2023-11-17T05:31:50.344635+0000 mon.smithi006 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi006.odexej(active, starting, since 0.00359275s) 2023-11-17T05:31:59.573 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.347447+0000 mon.smithi006 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:31:59.573 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.348162+0000 mon.smithi006 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:31:59.573 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.348830+0000 mon.smithi006 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.349288+0000 mon.smithi006 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.349708+0000 mon.smithi006 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi006.odexej", "id": "smithi006.odexej"}]: dispatch 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: cluster 2023-11-17T05:31:50.353894+0000 mon.smithi006 (mon.0) 16 : cluster [INF] Manager daemon smithi006.odexej is now available 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.373771+0000 mon.smithi006 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.380635+0000 mon.smithi006 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:31:59.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.384198+0000 mon.smithi006 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/trash_purge_schedule"}]: dispatch 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.388214+0000 mon.smithi006 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:50 smithi006 bash[26022]: audit 2023-11-17T05:31:50.391366+0000 mon.smithi006 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.6:0/4038919625' entity='mgr.smithi006.odexej' 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsid": "7d14d910-850a-11ee-8dbe-212e2dc638e7", 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:31:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 0 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "smithi006" 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:31:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:31:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:31:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "nfs", 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ], 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:31:35.037081+0000", 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: }, 2023-11-17T05:31:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:31:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:31:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:mgr is available 2023-11-17T05:31:59.588 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:52 smithi006 bash[26022]: cluster 2023-11-17T05:31:51.353942+0000 mon.smithi006 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi006.odexej(active, since 1.0129s) 2023-11-17T05:31:59.588 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:52 smithi006 bash[26022]: audit 2023-11-17T05:31:51.978399+0000 mon.smithi006 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.6:0/2510892854' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:31:59.588 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:53 smithi006 bash[26022]: cluster 2023-11-17T05:31:52.352399+0000 mon.smithi006 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi006.odexej(active, since 2s) 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [global] 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: fsid = 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-17T05:31:59.590 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [mgr] 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: [osd] 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-17T05:31:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:Enabling cephadm module... 2023-11-17T05:31:59.592 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:54 smithi006 bash[26022]: audit 2023-11-17T05:31:53.883061+0000 mon.smithi006 (mon.0) 25 : audit [INF] from='client.? 172.21.15.6:0/2174613751' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T05:31:59.592 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:56 smithi006 bash[26022]: audit 2023-11-17T05:31:55.988700+0000 mon.smithi006 (mon.0) 26 : audit [INF] from='client.? 172.21.15.6:0/643488246' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-17T05:31:59.592 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:57 smithi006 bash[26022]: audit 2023-11-17T05:31:56.044007+0000 mon.smithi006 (mon.0) 27 : audit [INF] from='client.? 172.21.15.6:0/643488246' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-17T05:31:59.592 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:57 smithi006 bash[26022]: cluster 2023-11-17T05:31:56.044108+0000 mon.smithi006 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi006.odexej(active, since 5s) 2023-11-17T05:31:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:31:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-17T05:31:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:31:59.623 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "active_name": "smithi006.odexej", 2023-11-17T05:31:59.623 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T05:31:59.624 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:31:59.624 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:31:58 smithi006 bash[26022]: audit 2023-11-17T05:31:58.072514+0000 mon.smithi006 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.6:0/1258468026' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T05:31:59.625 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for the mgr to restart... 2023-11-17T05:31:59.625 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mgr epoch 5... 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: cluster 2023-11-17T05:32:02.657253+0000 mon.smithi006 (mon.0) 30 : cluster [INF] Active manager daemon smithi006.odexej restarted 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: cluster 2023-11-17T05:32:02.657740+0000 mon.smithi006 (mon.0) 31 : cluster [INF] Activating manager daemon smithi006.odexej 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: cluster 2023-11-17T05:32:02.662983+0000 mon.smithi006 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: cluster 2023-11-17T05:32:02.663191+0000 mon.smithi006 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi006.odexej(active, starting, since 0.00560285s) 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.664941+0000 mon.smithi006 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.666497+0000 mon.smithi006 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi006.odexej", "id": "smithi006.odexej"}]: dispatch 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.667983+0000 mon.smithi006 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.668341+0000 mon.smithi006 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:32:02.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.668654+0000 mon.smithi006 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:32:02.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: cluster 2023-11-17T05:32:02.672314+0000 mon.smithi006 (mon.0) 39 : cluster [INF] Manager daemon smithi006.odexej is now available 2023-11-17T05:32:02.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:02 smithi006 bash[26022]: audit 2023-11-17T05:32:02.694522+0000 mon.smithi006 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:32:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-17T05:32:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T05:32:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:32:04.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: cephadm 2023-11-17T05:32:02.689402+0000 mgr.smithi006.odexej (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.817909+0000 mon.smithi006 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.819947+0000 mon.smithi006 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.823521+0000 mon.smithi006 (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.827813+0000 mon.smithi006 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.858587+0000 mon.smithi006 (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: audit 2023-11-17T05:32:02.863986+0000 mon.smithi006 (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/trash_purge_schedule"}]: dispatch 2023-11-17T05:32:04.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:03 smithi006 bash[26022]: cluster 2023-11-17T05:32:03.668291+0000 mon.smithi006 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi006.odexej(active, since 1.01068s) 2023-11-17T05:32:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:mgr epoch 5 is available 2023-11-17T05:32:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:Setting orchestrator backend to cephadm... 2023-11-17T05:32:05.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:04 smithi006 bash[26022]: audit 2023-11-17T05:32:03.667665+0000 mgr.smithi006.odexej (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T05:32:05.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:04 smithi006 bash[26022]: audit 2023-11-17T05:32:03.683960+0000 mgr.smithi006.odexej (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T05:32:06.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:05 smithi006 bash[26022]: cluster 2023-11-17T05:32:04.826694+0000 mon.smithi006 (mon.0) 48 : cluster [DBG] mgrmap e8: smithi006.odexej(active, since 2s) 2023-11-17T05:32:06.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:05 smithi006 bash[26022]: audit 2023-11-17T05:32:05.640444+0000 mon.smithi006 (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:06.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:05 smithi006 bash[26022]: audit 2023-11-17T05:32:05.659622+0000 mon.smithi006 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:32:07.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:07 smithi006 bash[26022]: audit 2023-11-17T05:32:05.633302+0000 mgr.smithi006.odexej (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: value unchanged 2023-11-17T05:32:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:Generating ssh key... 2023-11-17T05:32:08.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:08 smithi006 bash[26022]: audit 2023-11-17T05:32:07.519285+0000 mgr.smithi006.odexej (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:10.953 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:10 smithi006 bash[26022]: audit 2023-11-17T05:32:09.456558+0000 mgr.smithi006.odexej (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:10.953 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:10 smithi006 bash[26022]: cephadm 2023-11-17T05:32:09.456999+0000 mgr.smithi006.odexej (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2023-11-17T05:32:10.953 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:10 smithi006 bash[26022]: audit 2023-11-17T05:32:09.683680+0000 mon.smithi006 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:10.953 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:10 smithi006 bash[26022]: audit 2023-11-17T05:32:09.686287+0000 mon.smithi006 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf2zIlbH3XsOLUmu5nKRWZK3Aubo8uT08ui8PkxOXBPiW5tx/A1A3ieWHphdI5Q8ubiN2hhv4zT7fgDUn4XwVKzGJYvU989Sj5bbdDICBZirA24I/wT0opTTA4EmIJXmyW9v8LzaZN577GKl2Lj4pCs1P+U6rtUrOpdSOLUBpA7y/ltqExOIk1EdrT2Z/aTzQgYQ50lRDJ/TdsQ2BHxATgeuyPATDNgcL89jJ+TlPlZnB1mEVomH5JAILOQg+zMCDK9tJZ/JGeAGSXfG9him3CfuwhQu6YdM5i1prBQ+F9JXxQHQDL6x21pwk8YbYUaX1eQu4Dk0uErSqLIbYRrvQnrP7zYTTjND27qVS47KAptUxjuU229tgIaBmvZPcjdZkySTu2mawYe4/nqdClvFQ4/MDYPhWFgFtvmNTacWauDdbs/SMSDGIeeGhRjFlqbNDReqRJ2mPxZZ3nGcaBy+XXYeG3WNViZSuemr3foZdmveMXBkO22ry4EzdhdYxwCv8= ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:32:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-17T05:32:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:Adding key to root@localhost authorized_keys... 2023-11-17T05:32:12.096 INFO:teuthology.orchestra.run.smithi006.stderr:Adding host smithi006... 2023-11-17T05:32:12.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:12 smithi006 bash[26022]: audit 2023-11-17T05:32:11.539534+0000 mgr.smithi006.odexej (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:14.289 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Added host 'smithi006' with addr '172.21.15.6' 2023-11-17T05:32:14.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:14 smithi006 bash[26022]: audit 2023-11-17T05:32:13.480272+0000 mgr.smithi006.odexej (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying mon service with default placement... 2023-11-17T05:32:15.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:15 smithi006 bash[26022]: audit 2023-11-17T05:32:14.286414+0000 mon.smithi006 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:15.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:15 smithi006 bash[26022]: cephadm 2023-11-17T05:32:14.286872+0000 mgr.smithi006.odexej (mgr.14116) 10 : cephadm [INF] Added host smithi006 2023-11-17T05:32:15.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:15 smithi006 bash[26022]: audit 2023-11-17T05:32:14.296884+0000 mon.smithi006 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:32:16.775 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-17T05:32:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying mgr service with default placement... 2023-11-17T05:32:18.073 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:17 smithi006 bash[26022]: audit 2023-11-17T05:32:16.765918+0000 mgr.smithi006.odexej (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:18.073 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:17 smithi006 bash[26022]: cephadm 2023-11-17T05:32:16.769011+0000 mgr.smithi006.odexej (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-17T05:32:18.073 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:17 smithi006 bash[26022]: audit 2023-11-17T05:32:16.772322+0000 mon.smithi006 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:18.073 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:17 smithi006 bash[26022]: audit 2023-11-17T05:32:16.960037+0000 mon.smithi006 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:18.073 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:17 smithi006 bash[26022]: audit 2023-11-17T05:32:17.255393+0000 mon.smithi006 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:18.697 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-17T05:32:19.217 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying crash service with default placement... 2023-11-17T05:32:19.993 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:19 smithi006 bash[26022]: audit 2023-11-17T05:32:18.689181+0000 mgr.smithi006.odexej (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:19.993 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:19 smithi006 bash[26022]: cephadm 2023-11-17T05:32:18.690534+0000 mgr.smithi006.odexej (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-17T05:32:19.993 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:19 smithi006 bash[26022]: audit 2023-11-17T05:32:18.694571+0000 mon.smithi006 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:20.612 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-17T05:32:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying prometheus service with default placement... 2023-11-17T05:32:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:21 smithi006 bash[26022]: audit 2023-11-17T05:32:20.603276+0000 mgr.smithi006.odexej (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:21 smithi006 bash[26022]: cephadm 2023-11-17T05:32:20.605798+0000 mgr.smithi006.odexej (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2023-11-17T05:32:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:21 smithi006 bash[26022]: audit 2023-11-17T05:32:20.608873+0000 mon.smithi006 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:22.480 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-17T05:32:23.012 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying grafana service with default placement... 2023-11-17T05:32:23.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:23 smithi006 bash[26022]: audit 2023-11-17T05:32:22.398411+0000 mon.smithi006 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:23.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:23 smithi006 bash[26022]: audit 2023-11-17T05:32:22.469853+0000 mgr.smithi006.odexej (mgr.14116) 17 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:23.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:23 smithi006 bash[26022]: cephadm 2023-11-17T05:32:22.472146+0000 mgr.smithi006.odexej (mgr.14116) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-17T05:32:23.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:23 smithi006 bash[26022]: audit 2023-11-17T05:32:22.477508+0000 mon.smithi006 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:23.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:23 smithi006 bash[26022]: cluster 2023-11-17T05:32:22.669641+0000 mgr.smithi006.odexej (mgr.14116) 19 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:24.440 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-17T05:32:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying node-exporter service with default placement... 2023-11-17T05:32:25.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:25 smithi006 bash[26022]: audit 2023-11-17T05:32:24.430183+0000 mgr.smithi006.odexej (mgr.14116) 20 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:25.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:25 smithi006 bash[26022]: cephadm 2023-11-17T05:32:24.432459+0000 mgr.smithi006.odexej (mgr.14116) 21 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-17T05:32:25.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:25 smithi006 bash[26022]: audit 2023-11-17T05:32:24.437236+0000 mon.smithi006 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:25.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:25 smithi006 bash[26022]: cluster 2023-11-17T05:32:24.670135+0000 mgr.smithi006.odexej (mgr.14116) 22 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:26.282 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-17T05:32:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:Deploying alertmanager service with default placement... 2023-11-17T05:32:27.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:27 smithi006 bash[26022]: audit 2023-11-17T05:32:26.274240+0000 mgr.smithi006.odexej (mgr.14116) 23 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:27.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:27 smithi006 bash[26022]: cephadm 2023-11-17T05:32:26.276466+0000 mgr.smithi006.odexej (mgr.14116) 24 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-17T05:32:27.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:27 smithi006 bash[26022]: audit 2023-11-17T05:32:26.279711+0000 mon.smithi006 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:27.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:27 smithi006 bash[26022]: cluster 2023-11-17T05:32:26.670579+0000 mgr.smithi006.odexej (mgr.14116) 25 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-17T05:32:29.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:29 smithi006 bash[26022]: audit 2023-11-17T05:32:28.279359+0000 mgr.smithi006.odexej (mgr.14116) 26 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:32:29.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:29 smithi006 bash[26022]: cephadm 2023-11-17T05:32:28.280378+0000 mgr.smithi006.odexej (mgr.14116) 27 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-17T05:32:29.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:29 smithi006 bash[26022]: audit 2023-11-17T05:32:28.283887+0000 mon.smithi006 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.6:0/2183194566' entity='mgr.smithi006.odexej' 2023-11-17T05:32:29.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:29 smithi006 bash[26022]: cluster 2023-11-17T05:32:28.671138+0000 mgr.smithi006.odexej (mgr.14116) 28 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:31.841 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:31 smithi006 bash[26022]: audit 2023-11-17T05:32:30.530853+0000 mon.smithi006 (mon.0) 65 : audit [INF] from='client.? 172.21.15.6:0/1368709802' entity='client.admin' 2023-11-17T05:33:26.703 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:31 smithi006 bash[26022]: cluster 2023-11-17T05:32:30.671623+0000 mgr.smithi006.odexej (mgr.14116) 29 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.705 INFO:teuthology.orchestra.run.smithi006.stderr:Enabling the dashboard module... 2023-11-17T05:33:26.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:33 smithi006 bash[26022]: audit 2023-11-17T05:32:32.433141+0000 mon.smithi006 (mon.0) 66 : audit [INF] from='client.? 172.21.15.6:0/2357765169' entity='client.admin' 2023-11-17T05:33:26.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:33 smithi006 bash[26022]: cluster 2023-11-17T05:32:32.671870+0000 mgr.smithi006.odexej (mgr.14116) 30 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:34 smithi006 bash[26022]: audit 2023-11-17T05:32:34.390481+0000 mon.smithi006 (mon.0) 67 : audit [INF] from='client.? 172.21.15.6:0/2345728255' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-17T05:33:26.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:35 smithi006 bash[26022]: audit 2023-11-17T05:32:34.449613+0000 mon.smithi006 (mon.0) 68 : audit [INF] from='client.? 172.21.15.6:0/2345728255' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-17T05:33:26.706 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:35 smithi006 bash[26022]: cluster 2023-11-17T05:32:34.449801+0000 mon.smithi006 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi006.odexej(active, since 31s) 2023-11-17T05:33:26.734 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:33:26.734 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "epoch": 9, 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "active_name": "smithi006.odexej", 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for the mgr to restart... 2023-11-17T05:33:26.841 INFO:teuthology.orchestra.run.smithi006.stderr:Waiting for mgr epoch 9... 2023-11-17T05:33:26.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:37 smithi006 bash[26022]: audit 2023-11-17T05:32:36.633427+0000 mon.smithi006 (mon.0) 70 : audit [DBG] from='client.? 172.21.15.6:0/3452355947' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: cluster 2023-11-17T05:32:41.041642+0000 mon.smithi006 (mon.0) 71 : cluster [INF] Active manager daemon smithi006.odexej restarted 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: cluster 2023-11-17T05:32:41.042040+0000 mon.smithi006 (mon.0) 72 : cluster [INF] Activating manager daemon smithi006.odexej 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: cluster 2023-11-17T05:32:41.050439+0000 mon.smithi006 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: cluster 2023-11-17T05:32:41.050699+0000 mon.smithi006 (mon.0) 74 : cluster [DBG] mgrmap e10: smithi006.odexej(active, starting, since 0.0088007s) 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: audit 2023-11-17T05:32:41.052126+0000 mon.smithi006 (mon.0) 75 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: audit 2023-11-17T05:32:41.053735+0000 mon.smithi006 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi006.odexej", "id": "smithi006.odexej"}]: dispatch 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: audit 2023-11-17T05:32:41.054911+0000 mon.smithi006 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: audit 2023-11-17T05:32:41.055224+0000 mon.smithi006 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: audit 2023-11-17T05:32:41.055462+0000 mon.smithi006 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:33:26.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:41 smithi006 bash[26022]: cluster 2023-11-17T05:32:41.060221+0000 mon.smithi006 (mon.0) 80 : cluster [INF] Manager daemon smithi006.odexej is now available 2023-11-17T05:33:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: { 2023-11-17T05:33:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-11-17T05:33:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T05:33:26.869 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: } 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:42 smithi006 bash[26022]: audit 2023-11-17T05:32:41.200467+0000 mon.smithi006 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:42 smithi006 bash[26022]: audit 2023-11-17T05:32:41.206981+0000 mon.smithi006 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:42 smithi006 bash[26022]: audit 2023-11-17T05:32:41.252673+0000 mon.smithi006 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:42 smithi006 bash[26022]: audit 2023-11-17T05:32:41.258383+0000 mon.smithi006 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/trash_purge_schedule"}]: dispatch 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:42 smithi006 bash[26022]: cluster 2023-11-17T05:32:42.052487+0000 mon.smithi006 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi006.odexej(active, since 1.01058s) 2023-11-17T05:33:26.871 INFO:teuthology.orchestra.run.smithi006.stderr:mgr epoch 9 is available 2023-11-17T05:33:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:Generating a dashboard self-signed certificate... 2023-11-17T05:33:27.037 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:43 smithi006 bash[26022]: audit 2023-11-17T05:32:42.052242+0000 mgr.smithi006.odexej (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T05:33:27.037 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:43 smithi006 bash[26022]: audit 2023-11-17T05:32:42.066380+0000 mgr.smithi006.odexej (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T05:33:27.037 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-17T05:33:27.038 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:44 smithi006 bash[26022]: cluster 2023-11-17T05:32:43.110680+0000 mon.smithi006 (mon.0) 86 : cluster [DBG] mgrmap e12: smithi006.odexej(active, since 2s) 2023-11-17T05:33:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:Creating initial admin user... 2023-11-17T05:33:27.038 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:45 smithi006 bash[26022]: audit 2023-11-17T05:32:44.088454+0000 mgr.smithi006.odexej (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:33:27.038 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:45 smithi006 bash[26022]: audit 2023-11-17T05:32:44.128234+0000 mon.smithi006 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.038 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:45 smithi006 bash[26022]: audit 2023-11-17T05:32:44.134690+0000 mon.smithi006 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$dTsskSwHryWzMu66IRuHOOdK82mAIazhkSAUpOH23pKf.qgR4QloO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700199166, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-17T05:33:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:Fetching dashboard port number... 2023-11-17T05:33:27.039 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:47 smithi006 bash[26022]: audit 2023-11-17T05:32:45.986182+0000 mgr.smithi006.odexej (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:33:27.039 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:47 smithi006 bash[26022]: audit 2023-11-17T05:32:46.251124+0000 mon.smithi006 (mon.0) 89 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: 8443 2023-11-17T05:33:27.040 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:48 smithi006 bash[26022]: cluster 2023-11-17T05:32:47.257655+0000 mon.smithi006 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi006.odexej(active, since 6s) 2023-11-17T05:33:27.040 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:48 smithi006 bash[26022]: audit 2023-11-17T05:32:48.018568+0000 mon.smithi006 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.040 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:48 smithi006 bash[26022]: audit 2023-11-17T05:32:48.167989+0000 mon.smithi006 (mon.0) 92 : audit [DBG] from='client.? 172.21.15.6:0/2634303254' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-17T05:33:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:firewalld does not appear to be present 2023-11-17T05:33:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-17T05:33:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:Ceph Dashboard is now available at: 2023-11-17T05:33:27.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.042 INFO:teuthology.orchestra.run.smithi006.stderr: URL: https://smithi006.front.sepia.ceph.com:8443/ 2023-11-17T05:33:27.043 INFO:teuthology.orchestra.run.smithi006.stderr: User: admin 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr: Password: kbbonya4kp 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr:Enabling autotune for osd_memory_target 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr:Or, if you are only running a single cluster on this host: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: ceph telemetry on 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr:For more information see: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T05:33:27.257 INFO:teuthology.orchestra.run.smithi006.stderr:Bootstrap complete. 2023-11-17T05:33:27.258 INFO:tasks.cephadm:Fetching config... 2023-11-17T05:33:27.289 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:33:27.289 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-17T05:33:27.291 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:53 smithi006 bash[26022]: audit 2023-11-17T05:32:52.196732+0000 mon.smithi006 (mon.0) 93 : audit [INF] from='client.? 172.21.15.6:0/3944039507' entity='client.admin' 2023-11-17T05:33:27.291 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:59 smithi006 bash[26022]: audit 2023-11-17T05:32:58.655308+0000 mon.smithi006 (mon.0) 94 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:59 smithi006 bash[26022]: audit 2023-11-17T05:32:58.968491+0000 mon.smithi006 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:59 smithi006 bash[26022]: audit 2023-11-17T05:32:58.975931+0000 mon.smithi006 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:32:59 smithi006 bash[26022]: cephadm 2023-11-17T05:32:58.993245+0000 mgr.smithi006.odexej (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi006 on smithi006 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:02 smithi006 bash[26022]: cluster 2023-11-17T05:33:01.056766+0000 mgr.smithi006.odexej (mgr.14156) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:04 smithi006 bash[26022]: cluster 2023-11-17T05:33:03.057335+0000 mgr.smithi006.odexej (mgr.14156) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:06 smithi006 bash[26022]: cluster 2023-11-17T05:33:05.057902+0000 mgr.smithi006.odexej (mgr.14156) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.325 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:07 smithi006 bash[26022]: audit 2023-11-17T05:33:06.341497+0000 mon.smithi006 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.325 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:07 smithi006 bash[26022]: audit 2023-11-17T05:33:06.345862+0000 mon.smithi006 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:33:27.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:07 smithi006 bash[26022]: audit 2023-11-17T05:33:06.348380+0000 mon.smithi006 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-17T05:33:27.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:07 smithi006 bash[26022]: audit 2023-11-17T05:33:06.349457+0000 mon.smithi006 (mon.0) 100 : audit [DBG] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:33:27.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:07 smithi006 bash[26022]: cephadm 2023-11-17T05:33:06.350276+0000 mgr.smithi006.odexej (mgr.14156) 9 : cephadm [INF] Deploying daemon crash.smithi006 on smithi006 2023-11-17T05:33:27.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:08 smithi006 bash[26022]: cluster 2023-11-17T05:33:07.058442+0000 mgr.smithi006.odexej (mgr.14156) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:08.819255+0000 mon.smithi006 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: cluster 2023-11-17T05:33:09.058876+0000 mgr.smithi006.odexej (mgr.14156) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:09.077352+0000 mon.smithi006 (mon.0) 102 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:09.084212+0000 mon.smithi006 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:09.087828+0000 mon.smithi006 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:09.088759+0000 mgr.smithi006.odexej (mgr.14156) 12 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: audit 2023-11-17T05:33:09.096031+0000 mon.smithi006 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:09 smithi006 bash[26022]: cephadm 2023-11-17T05:33:09.119628+0000 mgr.smithi006.odexej (mgr.14156) 13 : cephadm [INF] Deploying daemon grafana.smithi006 on smithi006 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:12 smithi006 bash[26022]: cluster 2023-11-17T05:33:11.059238+0000 mgr.smithi006.odexej (mgr.14156) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:12 smithi006 bash[26022]: audit 2023-11-17T05:33:11.259406+0000 mon.smithi006 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:14 smithi006 bash[26022]: cluster 2023-11-17T05:33:13.059635+0000 mgr.smithi006.odexej (mgr.14156) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:16 smithi006 bash[26022]: cluster 2023-11-17T05:33:15.059999+0000 mgr.smithi006.odexej (mgr.14156) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:18 smithi006 bash[26022]: cluster 2023-11-17T05:33:17.060419+0000 mgr.smithi006.odexej (mgr.14156) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:20 smithi006 bash[26022]: cluster 2023-11-17T05:33:19.060954+0000 mgr.smithi006.odexej (mgr.14156) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:22 smithi006 bash[26022]: cluster 2023-11-17T05:33:21.061511+0000 mgr.smithi006.odexej (mgr.14156) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:24 smithi006 bash[26022]: cluster 2023-11-17T05:33:23.062045+0000 mgr.smithi006.odexej (mgr.14156) 20 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.329 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:26 smithi006 bash[26022]: cluster 2023-11-17T05:33:25.062474+0000 mgr.smithi006.odexej (mgr.14156) 21 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.329 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:26 smithi006 bash[26022]: audit 2023-11-17T05:33:26.252370+0000 mon.smithi006 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.329 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:26 smithi006 bash[26022]: audit 2023-11-17T05:33:26.266616+0000 mon.smithi006 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.335 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-17T05:33:27.335 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:33:27.335 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-17T05:33:27.383 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-17T05:33:27.383 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:33:27.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/keyring of=/dev/stdout 2023-11-17T05:33:27.440 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-17T05:33:27.440 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:33:27.441 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-17T05:33:27.487 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-17T05:33:27.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf2zIlbH3XsOLUmu5nKRWZK3Aubo8uT08ui8PkxOXBPiW5tx/A1A3ieWHphdI5Q8ubiN2hhv4zT7fgDUn4XwVKzGJYvU989Sj5bbdDICBZirA24I/wT0opTTA4EmIJXmyW9v8LzaZN577GKl2Lj4pCs1P+U6rtUrOpdSOLUBpA7y/ltqExOIk1EdrT2Z/aTzQgYQ50lRDJ/TdsQ2BHxATgeuyPATDNgcL89jJ+TlPlZnB1mEVomH5JAILOQg+zMCDK9tJZ/JGeAGSXfG9him3CfuwhQu6YdM5i1prBQ+F9JXxQHQDL6x21pwk8YbYUaX1eQu4Dk0uErSqLIbYRrvQnrP7zYTTjND27qVS47KAptUxjuU229tgIaBmvZPcjdZkySTu2mawYe4/nqdClvFQ4/MDYPhWFgFtvmNTacWauDdbs/SMSDGIeeGhRjFlqbNDReqRJ2mPxZZ3nGcaBy+XXYeG3WNViZSuemr3foZdmveMXBkO22ry4EzdhdYxwCv8= ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T05:33:27.532 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:27 smithi006 bash[26022]: cephadm 2023-11-17T05:33:26.257337+0000 mgr.smithi006.odexej (mgr.14156) 22 : cephadm [INF] Deploying daemon node-exporter.smithi006 on smithi006 2023-11-17T05:33:27.532 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:27 smithi006 bash[26022]: audit 2023-11-17T05:33:26.802250+0000 mon.smithi006 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' 2023-11-17T05:33:27.532 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:27 smithi006 bash[26022]: audit 2023-11-17T05:33:26.805643+0000 mon.smithi006 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-17T05:33:27.553 INFO:teuthology.orchestra.run.smithi006.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf2zIlbH3XsOLUmu5nKRWZK3Aubo8uT08ui8PkxOXBPiW5tx/A1A3ieWHphdI5Q8ubiN2hhv4zT7fgDUn4XwVKzGJYvU989Sj5bbdDICBZirA24I/wT0opTTA4EmIJXmyW9v8LzaZN577GKl2Lj4pCs1P+U6rtUrOpdSOLUBpA7y/ltqExOIk1EdrT2Z/aTzQgYQ50lRDJ/TdsQ2BHxATgeuyPATDNgcL89jJ+TlPlZnB1mEVomH5JAILOQg+zMCDK9tJZ/JGeAGSXfG9him3CfuwhQu6YdM5i1prBQ+F9JXxQHQDL6x21pwk8YbYUaX1eQu4Dk0uErSqLIbYRrvQnrP7zYTTjND27qVS47KAptUxjuU229tgIaBmvZPcjdZkySTu2mawYe4/nqdClvFQ4/MDYPhWFgFtvmNTacWauDdbs/SMSDGIeeGhRjFlqbNDReqRJ2mPxZZ3nGcaBy+XXYeG3WNViZSuemr3foZdmveMXBkO22ry4EzdhdYxwCv8= ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:33:27.560 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf2zIlbH3XsOLUmu5nKRWZK3Aubo8uT08ui8PkxOXBPiW5tx/A1A3ieWHphdI5Q8ubiN2hhv4zT7fgDUn4XwVKzGJYvU989Sj5bbdDICBZirA24I/wT0opTTA4EmIJXmyW9v8LzaZN577GKl2Lj4pCs1P+U6rtUrOpdSOLUBpA7y/ltqExOIk1EdrT2Z/aTzQgYQ50lRDJ/TdsQ2BHxATgeuyPATDNgcL89jJ+TlPlZnB1mEVomH5JAILOQg+zMCDK9tJZ/JGeAGSXfG9him3CfuwhQu6YdM5i1prBQ+F9JXxQHQDL6x21pwk8YbYUaX1eQu4Dk0uErSqLIbYRrvQnrP7zYTTjND27qVS47KAptUxjuU229tgIaBmvZPcjdZkySTu2mawYe4/nqdClvFQ4/MDYPhWFgFtvmNTacWauDdbs/SMSDGIeeGhRjFlqbNDReqRJ2mPxZZ3nGcaBy+XXYeG3WNViZSuemr3foZdmveMXBkO22ry4EzdhdYxwCv8= ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T05:33:27.585 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCf2zIlbH3XsOLUmu5nKRWZK3Aubo8uT08ui8PkxOXBPiW5tx/A1A3ieWHphdI5Q8ubiN2hhv4zT7fgDUn4XwVKzGJYvU989Sj5bbdDICBZirA24I/wT0opTTA4EmIJXmyW9v8LzaZN577GKl2Lj4pCs1P+U6rtUrOpdSOLUBpA7y/ltqExOIk1EdrT2Z/aTzQgYQ50lRDJ/TdsQ2BHxATgeuyPATDNgcL89jJ+TlPlZnB1mEVomH5JAILOQg+zMCDK9tJZ/JGeAGSXfG9him3CfuwhQu6YdM5i1prBQ+F9JXxQHQDL6x21pwk8YbYUaX1eQu4Dk0uErSqLIbYRrvQnrP7zYTTjND27qVS47KAptUxjuU229tgIaBmvZPcjdZkySTu2mawYe4/nqdClvFQ4/MDYPhWFgFtvmNTacWauDdbs/SMSDGIeeGhRjFlqbNDReqRJ2mPxZZ3nGcaBy+XXYeG3WNViZSuemr3foZdmveMXBkO22ry4EzdhdYxwCv8= ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:33:27.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-17T05:33:28.384 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:28 smithi006 bash[26022]: cluster 2023-11-17T05:33:27.062903+0000 mgr.smithi006.odexej (mgr.14156) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:28.384 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:28 smithi006 bash[26022]: audit 2023-11-17T05:33:27.808342+0000 mon.smithi006 (mon.0) 111 : audit [INF] from='mgr.14156 172.21.15.6:0/2968240898' entity='mgr.smithi006.odexej' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-17T05:33:28.384 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:28 smithi006 bash[26022]: cluster 2023-11-17T05:33:27.808398+0000 mon.smithi006 (mon.0) 112 : cluster [DBG] mgrmap e14: smithi006.odexej(active, since 46s) 2023-11-17T05:33:33.828 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-17T05:33:33.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-17T05:33:34.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:33 smithi006 bash[26022]: audit 2023-11-17T05:33:32.886544+0000 mon.smithi006 (mon.0) 113 : audit [INF] from='client.? 172.21.15.6:0/44675331' entity='client.admin' 2023-11-17T05:33:34.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: cluster 2023-11-17T05:33:34.469223+0000 mon.smithi006 (mon.0) 114 : cluster [INF] Active manager daemon smithi006.odexej restarted 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: cluster 2023-11-17T05:33:34.469516+0000 mon.smithi006 (mon.0) 115 : cluster [INF] Activating manager daemon smithi006.odexej 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: cluster 2023-11-17T05:33:34.474047+0000 mon.smithi006 (mon.0) 116 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: cluster 2023-11-17T05:33:34.474220+0000 mon.smithi006 (mon.0) 117 : cluster [DBG] mgrmap e15: smithi006.odexej(active, starting, since 0.00481009s) 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.475315+0000 mon.smithi006 (mon.0) 118 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.477333+0000 mon.smithi006 (mon.0) 119 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi006.odexej", "id": "smithi006.odexej"}]: dispatch 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.478348+0000 mon.smithi006 (mon.0) 120 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.478519+0000 mon.smithi006 (mon.0) 121 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.478744+0000 mon.smithi006 (mon.0) 122 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: cluster 2023-11-17T05:33:34.481546+0000 mon.smithi006 (mon.0) 123 : cluster [INF] Manager daemon smithi006.odexej is now available 2023-11-17T05:33:34.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.625182+0000 mon.smithi006 (mon.0) 124 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:33:34.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.626996+0000 mon.smithi006 (mon.0) 125 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:33:34.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.666743+0000 mon.smithi006 (mon.0) 126 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:34.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.684179+0000 mon.smithi006 (mon.0) 127 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:33:34.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:34 smithi006 bash[26022]: audit 2023-11-17T05:33:34.688983+0000 mon.smithi006 (mon.0) 128 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.odexej/trash_purge_schedule"}]: dispatch 2023-11-17T05:33:36.776 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:36 smithi006 bash[26022]: cluster 2023-11-17T05:33:35.476440+0000 mon.smithi006 (mon.0) 129 : cluster [DBG] mgrmap e16: smithi006.odexej(active, since 1.00703s) 2023-11-17T05:33:38.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:38 smithi006 bash[26022]: cluster 2023-11-17T05:33:37.105786+0000 mon.smithi006 (mon.0) 130 : cluster [DBG] mgrmap e17: smithi006.odexej(active, since 2s) 2023-11-17T05:33:38.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:38 smithi006 bash[26022]: audit 2023-11-17T05:33:37.790949+0000 mgr.smithi006.odexej (mgr.14176) 1 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:33:38.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:38 smithi006 bash[26022]: audit 2023-11-17T05:33:37.794929+0000 mon.smithi006 (mon.0) 131 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:38.568 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2023-11-17T05:33:38.568 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:33:38.568 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2023-11-17T05:33:38.578 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:33:38.578 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:33:38.628 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2023-11-17T05:33:38.628 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi156 2023-11-17T05:33:42.191 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.021295+0000 mon.smithi006 (mon.0) 132 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.316688+0000 mon.smithi006 (mon.0) 133 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.318064+0000 mon.smithi006 (mon.0) 134 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.319162+0000 mon.smithi006 (mon.0) 135 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: cephadm 2023-11-17T05:33:41.321265+0000 mgr.smithi006.odexej (mgr.14176) 2 : cephadm [INF] Updating smithi006:/etc/ceph/ceph.conf 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: cephadm 2023-11-17T05:33:41.354884+0000 mgr.smithi006.odexej (mgr.14176) 3 : cephadm [INF] Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.389322+0000 mon.smithi006 (mon.0) 136 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: audit 2023-11-17T05:33:41.396511+0000 mon.smithi006 (mon.0) 137 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:42.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:42 smithi006 bash[26022]: cephadm 2023-11-17T05:33:41.422521+0000 mgr.smithi006.odexej (mgr.14176) 4 : cephadm [INF] Deploying daemon prometheus.smithi006 on smithi006 2023-11-17T05:33:55.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:54 smithi006 bash[26022]: audit 2023-11-17T05:33:53.933558+0000 mon.smithi006 (mon.0) 138 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:55.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:54 smithi006 bash[26022]: audit 2023-11-17T05:33:53.935167+0000 mon.smithi006 (mon.0) 139 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:33:55.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:54 smithi006 bash[26022]: cluster 2023-11-17T05:33:54.480028+0000 mgr.smithi006.odexej (mgr.14176) 5 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:55.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:54 smithi006 bash[26022]: audit 2023-11-17T05:33:54.672019+0000 mon.smithi006 (mon.0) 140 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:33:57.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:57 smithi006 bash[26022]: cluster 2023-11-17T05:33:56.480493+0000 mgr.smithi006.odexej (mgr.14176) 6 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:59.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:33:59 smithi006 bash[26022]: cluster 2023-11-17T05:33:58.480948+0000 mgr.smithi006.odexej (mgr.14176) 7 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:01.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.962522+0000 mon.smithi006 (mon.0) 141 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:01.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.964374+0000 mon.smithi006 (mon.0) 142 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:01.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.966040+0000 mon.smithi006 (mon.0) 143 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:34:01.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.970917+0000 mon.smithi006 (mon.0) 144 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: cephadm 2023-11-17T05:33:59.976415+0000 mgr.smithi006.odexej (mgr.14176) 8 : cephadm [INF] Reconfiguring mon.smithi006 (unknown last config time)... 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.976830+0000 mon.smithi006 (mon.0) 145 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.978050+0000 mon.smithi006 (mon.0) 146 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: audit 2023-11-17T05:33:59.979358+0000 mon.smithi006 (mon.0) 147 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: cephadm 2023-11-17T05:33:59.980592+0000 mgr.smithi006.odexej (mgr.14176) 9 : cephadm [INF] Reconfiguring daemon mon.smithi006 on smithi006 2023-11-17T05:34:01.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:00 smithi006 bash[26022]: cluster 2023-11-17T05:34:00.481445+0000 mgr.smithi006.odexej (mgr.14176) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: audit 2023-11-17T05:34:01.894914+0000 mon.smithi006 (mon.0) 148 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: cephadm 2023-11-17T05:34:01.896232+0000 mgr.smithi006.odexej (mgr.14176) 11 : cephadm [INF] Reconfiguring mgr.smithi006.odexej (unknown last config time)... 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: audit 2023-11-17T05:34:01.896828+0000 mon.smithi006 (mon.0) 149 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi006.odexej", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: audit 2023-11-17T05:34:01.898535+0000 mon.smithi006 (mon.0) 150 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: audit 2023-11-17T05:34:01.900300+0000 mon.smithi006 (mon.0) 151 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: cephadm 2023-11-17T05:34:01.901368+0000 mgr.smithi006.odexej (mgr.14176) 12 : cephadm [INF] Reconfiguring daemon mgr.smithi006.odexej on smithi006 2023-11-17T05:34:03.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:02 smithi006 bash[26022]: cluster 2023-11-17T05:34:02.481944+0000 mgr.smithi006.odexej (mgr.14176) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:04.128 INFO:teuthology.orchestra.run.smithi156.stdout:Added host 'smithi156' with addr '172.21.15.156' 2023-11-17T05:34:04.801 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-17T05:34:04.905 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:04 smithi006 bash[26022]: audit 2023-11-17T05:34:03.287042+0000 mgr.smithi006.odexej (mgr.14176) 14 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:34:04.905 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:04 smithi006 bash[26022]: audit 2023-11-17T05:34:03.566398+0000 mon.smithi006 (mon.0) 152 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:04.905 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:04 smithi006 bash[26022]: cephadm 2023-11-17T05:34:03.568221+0000 mgr.smithi006.odexej (mgr.14176) 15 : cephadm [INF] Reconfiguring alertmanager.smithi006 (dependencies changed)... 2023-11-17T05:34:04.905 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:04 smithi006 bash[26022]: cephadm 2023-11-17T05:34:03.579299+0000 mgr.smithi006.odexej (mgr.14176) 16 : cephadm [INF] Reconfiguring daemon alertmanager.smithi006 on smithi006 2023-11-17T05:34:04.906 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:04 smithi006 bash[26022]: audit 2023-11-17T05:34:04.122890+0000 mon.smithi006 (mon.0) 153 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:05.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:05 smithi006 bash[26022]: cephadm 2023-11-17T05:34:04.123408+0000 mgr.smithi006.odexej (mgr.14176) 17 : cephadm [INF] Added host smithi156 2023-11-17T05:34:05.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:05 smithi006 bash[26022]: cluster 2023-11-17T05:34:04.482478+0000 mgr.smithi006.odexej (mgr.14176) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:07.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:07 smithi006 bash[26022]: cluster 2023-11-17T05:34:06.482940+0000 mgr.smithi006.odexej (mgr.14176) 19 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:07.516 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-17T05:34:07.516 INFO:teuthology.orchestra.run.smithi156.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2023-11-17T05:34:08.022 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-17T05:34:08.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-17T05:34:08.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:08 smithi006 bash[26022]: audit 2023-11-17T05:34:07.512512+0000 mgr.smithi006.odexej (mgr.14176) 20 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:34:16.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:08 smithi006 bash[26022]: audit 2023-11-17T05:34:08.017850+0000 mon.smithi006 (mon.0) 154 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:09 smithi006 bash[26022]: cephadm 2023-11-17T05:34:08.019573+0000 mgr.smithi006.odexej (mgr.14176) 21 : cephadm [INF] Reconfiguring grafana.smithi006 (dependencies changed)... 2023-11-17T05:34:16.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:09 smithi006 bash[26022]: cephadm 2023-11-17T05:34:08.067976+0000 mgr.smithi006.odexej (mgr.14176) 22 : cephadm [INF] Reconfiguring daemon grafana.smithi006 on smithi006 2023-11-17T05:34:16.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:09 smithi006 bash[26022]: cluster 2023-11-17T05:34:08.483387+0000 mgr.smithi006.odexej (mgr.14176) 23 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:11 smithi006 bash[26022]: cluster 2023-11-17T05:34:10.483797+0000 mgr.smithi006.odexej (mgr.14176) 24 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.171 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2023-11-17T05:34:16.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:12 smithi006 bash[26022]: audit 2023-11-17T05:34:12.221043+0000 mon.smithi006 (mon.0) 155 : audit [INF] from='client.? 172.21.15.6:0/1162214365' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-17T05:34:16.172 INFO:tasks.cephadm:Adding mon.smithi006 on smithi006 2023-11-17T05:34:16.173 INFO:tasks.cephadm:Adding mon.smithi156 on smithi156 2023-11-17T05:34:16.173 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi006:172.21.15.6=smithi006;smithi156:172.21.15.156=smithi156' 2023-11-17T05:34:16.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:13 smithi006 bash[26022]: cluster 2023-11-17T05:34:12.484209+0000 mgr.smithi006.odexej (mgr.14176) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:13 smithi006 bash[26022]: audit 2023-11-17T05:34:12.546314+0000 mon.smithi006 (mon.0) 156 : audit [INF] from='client.? 172.21.15.6:0/1162214365' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-17T05:34:16.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:13 smithi006 bash[26022]: cluster 2023-11-17T05:34:12.546465+0000 mon.smithi006 (mon.0) 157 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.863173+0000 mon.smithi006 (mon.0) 158 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.866198+0000 mon.smithi006 (mon.0) 159 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.868520+0000 mon.smithi006 (mon.0) 160 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.874953+0000 mon.smithi006 (mon.0) 161 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.882148+0000 mon.smithi006 (mon.0) 162 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.894763+0000 mon.smithi006 (mon.0) 163 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.903690+0000 mon.smithi006 (mon.0) 164 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.909611+0000 mon.smithi006 (mon.0) 165 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.914109+0000 mon.smithi006 (mon.0) 166 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.921799+0000 mon.smithi006 (mon.0) 167 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:13.928516+0000 mon.smithi006 (mon.0) 168 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:34:16.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:14 smithi006 bash[26022]: audit 2023-11-17T05:34:14.522998+0000 mon.smithi006 (mon.0) 169 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.866823+0000 mgr.smithi006.odexej (mgr.14176) 26 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.869081+0000 mgr.smithi006.odexej (mgr.14176) 27 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.882937+0000 mgr.smithi006.odexej (mgr.14176) 28 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.895729+0000 mgr.smithi006.odexej (mgr.14176) 29 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.910314+0000 mgr.smithi006.odexej (mgr.14176) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:13.914739+0000 mgr.smithi006.odexej (mgr.14176) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: cluster 2023-11-17T05:34:14.484829+0000 mgr.smithi006.odexej (mgr.14176) 32 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:15 smithi006 bash[26022]: audit 2023-11-17T05:34:15.820498+0000 mon.smithi006 (mon.0) 170 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:17.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:17 smithi006 bash[26022]: cluster 2023-11-17T05:34:16.485385+0000 mgr.smithi006.odexej (mgr.14176) 33 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:19.892 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2023-11-17T05:34:19.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:19 smithi006 bash[26022]: cluster 2023-11-17T05:34:18.485869+0000 mgr.smithi006.odexej (mgr.14176) 34 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:20.601 DEBUG:teuthology.orchestra.run.smithi156:mon.smithi156> sudo journalctl -f -n 0 -u ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi156.service 2023-11-17T05:34:20.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:20.603 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:21.096 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2023-11-17T05:34:21.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:20 smithi006 bash[26022]: audit 2023-11-17T05:34:19.877195+0000 mgr.smithi006.odexej (mgr.14176) 35 : audit [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi006:172.21.15.6=smithi006;smithi156:172.21.15.156=smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:34:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:20 smithi006 bash[26022]: cephadm 2023-11-17T05:34:19.882863+0000 mgr.smithi006.odexej (mgr.14176) 36 : cephadm [INF] Saving service mon spec with placement smithi006:172.21.15.6=smithi006;smithi156:172.21.15.156=smithi156;count:2 2023-11-17T05:34:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:20 smithi006 bash[26022]: audit 2023-11-17T05:34:19.888362+0000 mon.smithi006 (mon.0) 171 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:20 smithi006 bash[26022]: audit 2023-11-17T05:34:20.016778+0000 mon.smithi006 (mon.0) 172 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:20 smithi006 bash[26022]: cluster 2023-11-17T05:34:20.486319+0000 mgr.smithi006.odexej (mgr.14176) 37 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:23.810 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-17T05:34:23.810 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:31:31.143165Z","created":"2023-11-17T05:31:31.143165Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:23.813 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: cluster 2023-11-17T05:34:22.486771+0000 mgr.smithi006.odexej (mgr.14176) 38 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:22.747945+0000 mon.smithi006 (mon.0) 173 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.088014+0000 mon.smithi006 (mon.0) 174 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.089777+0000 mon.smithi006 (mon.0) 175 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.092122+0000 mon.smithi006 (mon.0) 176 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.094001+0000 mon.smithi006 (mon.0) 177 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.178322+0000 mon.smithi006 (mon.0) 178 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.184635+0000 mon.smithi006 (mon.0) 179 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.187330+0000 mon.smithi006 (mon.0) 180 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi156", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.191791+0000 mon.smithi006 (mon.0) 181 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi156", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-17T05:34:23.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:23 smithi006 bash[26022]: audit 2023-11-17T05:34:23.193420+0000 mon.smithi006 (mon.0) 182 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:25.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:24 smithi006 bash[26022]: cephadm 2023-11-17T05:34:23.096787+0000 mgr.smithi006.odexej (mgr.14176) 39 : cephadm [INF] Updating smithi156:/etc/ceph/ceph.conf 2023-11-17T05:34:25.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:24 smithi006 bash[26022]: cephadm 2023-11-17T05:34:23.128317+0000 mgr.smithi006.odexej (mgr.14176) 40 : cephadm [INF] Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:34:25.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:24 smithi006 bash[26022]: cephadm 2023-11-17T05:34:23.195011+0000 mgr.smithi006.odexej (mgr.14176) 41 : cephadm [INF] Deploying daemon crash.smithi156 on smithi156 2023-11-17T05:34:25.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:24 smithi006 bash[26022]: audit 2023-11-17T05:34:23.807094+0000 mon.smithi006 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.156:0/1677361284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:25.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:25.632 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:26.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:25 smithi006 bash[26022]: cluster 2023-11-17T05:34:24.487213+0000 mgr.smithi006.odexej (mgr.14176) 42 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: audit 2023-11-17T05:34:26.068352+0000 mon.smithi006 (mon.0) 184 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: audit 2023-11-17T05:34:26.073617+0000 mon.smithi006 (mon.0) 185 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi156.fphsnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:34:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: audit 2023-11-17T05:34:26.078454+0000 mon.smithi006 (mon.0) 186 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi156.fphsnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-17T05:34:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: audit 2023-11-17T05:34:26.080306+0000 mon.smithi006 (mon.0) 187 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:34:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: audit 2023-11-17T05:34:26.082059+0000 mon.smithi006 (mon.0) 188 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: cephadm 2023-11-17T05:34:26.083715+0000 mgr.smithi006.odexej (mgr.14176) 43 : cephadm [INF] Deploying daemon mgr.smithi156.fphsnp on smithi156 2023-11-17T05:34:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:27 smithi006 bash[26022]: cluster 2023-11-17T05:34:26.487670+0000 mgr.smithi006.odexej (mgr.14176) 44 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:30.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: cluster 2023-11-17T05:34:28.488121+0000 mgr.smithi006.odexej (mgr.14176) 45 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:30.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: audit 2023-11-17T05:34:28.811995+0000 mon.smithi006 (mon.0) 189 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:30.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: audit 2023-11-17T05:34:28.817025+0000 mon.smithi006 (mon.0) 190 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:34:30.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: audit 2023-11-17T05:34:28.818705+0000 mon.smithi006 (mon.0) 191 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:30.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: cephadm 2023-11-17T05:34:28.820424+0000 mgr.smithi006.odexej (mgr.14176) 46 : cephadm [INF] Deploying daemon mon.smithi156 on smithi156 2023-11-17T05:34:30.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:29 smithi006 bash[26022]: audit 2023-11-17T05:34:29.691623+0000 mon.smithi006 (mon.0) 192 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:30.752 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-17T05:34:30.752 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:31:31.143165Z","created":"2023-11-17T05:31:31.143165Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:30.758 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2023-11-17T05:34:31.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:30 smithi006 bash[26022]: audit 2023-11-17T05:34:30.749636+0000 mon.smithi006 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.156:0/329875772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:32.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:31 smithi006 bash[26022]: cluster 2023-11-17T05:34:30.488506+0000 mgr.smithi006.odexej (mgr.14176) 47 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:32.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:32.669 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:33.374 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:33 smithi156 systemd[1]: Started Ceph mon.smithi156 for 7d14d910-850a-11ee-8dbe-212e2dc638e7. 2023-11-17T05:34:34.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:33 smithi006 bash[26022]: cluster 2023-11-17T05:34:32.488930+0000 mgr.smithi006.odexej (mgr.14176) 48 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:34.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:33 smithi006 bash[26022]: audit 2023-11-17T05:34:33.111171+0000 mon.smithi006 (mon.0) 194 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:34.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:33 smithi006 bash[26022]: audit 2023-11-17T05:34:33.661590+0000 mon.smithi006 (mon.0) 195 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:34.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:33 smithi006 bash[26022]: audit 2023-11-17T05:34:33.665459+0000 mon.smithi006 (mon.0) 196 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.675+0000 7fe62a81f880 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.675+0000 7fe62a81f880 0 ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable), process ceph-mon, pid 7 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.675+0000 7fe62a81f880 0 pidfile_write: ignore empty --pid-file 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 0 load: jerasure load: lrc load: isa 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T05:34:34.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Compile date Nov 14 2023 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: DB SUMMARY 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: CURRENT file: CURRENT 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi156/store.db dir, Total Num: 0, files: 2023-11-17T05:34:34.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi156/store.db: 000003.log size: 511 ; 2023-11-17T05:34:34.811 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.error_if_exists: 0 2023-11-17T05:34:34.811 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.create_if_missing: 0 2023-11-17T05:34:34.811 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T05:34:34.811 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.env: 0x560cd65eb080 2023-11-17T05:34:34.811 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.fs: Posix File System 2023-11-17T05:34:34.812 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.info_log: 0x560cd7614580 2023-11-17T05:34:34.812 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T05:34:34.812 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.statistics: (nil) 2023-11-17T05:34:34.812 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.use_fsync: 0 2023-11-17T05:34:34.813 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T05:34:34.813 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T05:34:34.813 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T05:34:34.813 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T05:34:34.813 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T05:34:34.814 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T05:34:34.814 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T05:34:34.814 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T05:34:34.816 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T05:34:34.817 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T05:34:34.817 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T05:34:34.817 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.db_log_dir: 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi156/store.db 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T05:34:34.818 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T05:34:34.819 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T05:34:34.819 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T05:34:34.819 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T05:34:34.819 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.write_buffer_manager: 0x560cd8262480 2023-11-17T05:34:34.830 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T05:34:34.831 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T05:34:34.831 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T05:34:34.831 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T05:34:34.831 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T05:34:34.831 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.unordered_write: 0 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T05:34:34.832 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.row_cache: None 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.wal_filter: None 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T05:34:34.833 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T05:34:34.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T05:34:34.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.two_write_queues: 0 2023-11-17T05:34:34.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T05:34:34.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.atomic_flush: 0 2023-11-17T05:34:34.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T05:34:34.835 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.679+0000 7fe62a81f880 4 rocksdb: Options.max_open_files: -1 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T05:34:34.836 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T05:34:34.844 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T05:34:34.845 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T05:34:34.845 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Compression algorithms supported: 2023-11-17T05:34:34.845 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T05:34:34.845 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kZSTD supported: 0 2023-11-17T05:34:34.845 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kXpressCompression supported: 0 2023-11-17T05:34:34.846 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T05:34:34.855 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T05:34:34.855 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T05:34:34.855 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kZlibCompression supported: 1 2023-11-17T05:34:34.855 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T05:34:34.856 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T05:34:34.856 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi156/store.db/MANIFEST-000001 2023-11-17T05:34:34.856 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T05:34:34.856 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T05:34:34.866 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.merge_operator: 2023-11-17T05:34:34.869 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_filter: None 2023-11-17T05:34:34.869 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T05:34:34.869 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560cd75b7d30) 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: cache_index_and_filter_blocks: 1 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T05:34:34.870 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: pin_top_level_index_and_filter: 1 2023-11-17T05:34:34.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: index_type: 0 2023-11-17T05:34:34.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: data_block_index_type: 0 2023-11-17T05:34:34.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: index_shortening: 1 2023-11-17T05:34:34.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: data_block_hash_table_util_ratio: 0.750000 2023-11-17T05:34:34.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: hash_index_allow_collision: 1 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: checksum: 1 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: no_block_cache: 0 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_cache: 0x560cd75eef10 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_cache_name: BinnedLRUCache 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_cache_options: 2023-11-17T05:34:34.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: capacity : 536870912 2023-11-17T05:34:34.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: num_shard_bits : 4 2023-11-17T05:34:34.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: strict_capacity_limit : 0 2023-11-17T05:34:34.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: high_pri_pool_ratio: 0.000 2023-11-17T05:34:34.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_cache_compressed: (nil) 2023-11-17T05:34:34.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: persistent_cache: (nil) 2023-11-17T05:34:34.875 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_size: 4096 2023-11-17T05:34:34.875 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_size_deviation: 10 2023-11-17T05:34:34.875 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_restart_interval: 16 2023-11-17T05:34:34.875 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: index_block_restart_interval: 1 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: metadata_block_size: 4096 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: partition_filters: 0 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: use_delta_encoding: 1 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: whole_key_filtering: 1 2023-11-17T05:34:34.876 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: verify_compression: 0 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: read_amp_bytes_per_bit: 0 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: format_version: 2 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: enable_index_compression: 1 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: block_align: 0 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T05:34:34.900 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression: NoCompression 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.num_levels: 7 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:34:34.901 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T05:34:34.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T05:34:34.903 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T05:34:34.904 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.table_properties_collectors: 2023-11-17T05:34:34.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T05:34:34.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T05:34:34.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.bloom_locality: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.ttl: 2592000 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi156/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-17T05:34:34.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199274685908, "job": 1, "event": "recovery_started", "log_files": [3]} 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199274686468, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700199274, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700199274687038, "job": 1, "event": "recovery_finished"} 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.683+0000 7fe62a81f880 4 rocksdb: DB pointer 0x560cd76b7800 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe611625700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe611625700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** DB Stats ** 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T05:34:34.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** Compaction Stats [default] ** 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** Compaction Stats [default] ** 2023-11-17T05:34:34.925 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Cumulative compaction: 0.00 GB write, 0.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Interval compaction: 0.00 GB write, 0.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:34:34.926 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** Compaction Stats [default] ** 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** Compaction Stats [default] ** 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:34:34.927 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 0.00 0.00 1 0.001 0 0 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Cumulative compaction: 0.00 GB write, 0.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe62a81f880 0 mon.smithi156 does not exist in monmap, will attempt to join an existing cluster 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe62a81f880 0 using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-11-17T05:34:34.928 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe62a81f880 0 starting mon.smithi156 rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi156 fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.687+0000 7fe62a81f880 1 mon.smithi156@-1(???) e0 preinit fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 0 mon.smithi156@-1(synchronizing).mds e1 new map 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 0 mon.smithi156@-1(synchronizing).mds e1 print_map 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: e1 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: legacy client fscid: -1 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: No filesystems configured 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-17T05:34:34.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 0 mon.smithi156@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 0 mon.smithi156@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 0 mon.smithi156@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: cluster 2023-11-17T05:34:32.488930+0000 mgr.smithi006.odexej (mgr.14176) 48 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: audit 2023-11-17T05:34:33.111171+0000 mon.smithi006 (mon.0) 194 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: audit 2023-11-17T05:34:33.661590+0000 mon.smithi006 (mon.0) 195 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: audit 2023-11-17T05:34:33.665459+0000 mon.smithi006 (mon.0) 196 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.739+0000 7fe613e2a700 1 mon.smithi156@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-11-17T05:34:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:34 smithi156 bash[29655]: debug 2023-11-17T05:34:34.747+0000 7fe613e2a700 10 mon.smithi156@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:34.489416+0000 mgr.smithi006.odexej (mgr.14176) 50 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:34.758701+0000 mon.smithi006 (mon.0) 199 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:34.759071+0000 mon.smithi006 (mon.0) 200 : cluster [INF] mon.smithi006 calling monitor election 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:34.780822+0000 mon.smithi006 (mon.0) 201 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:35.752424+0000 mon.smithi006 (mon.0) 202 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:36.489798+0000 mgr.smithi006.odexej (mgr.14176) 51 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:36.752432+0000 mon.smithi006 (mon.0) 203 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:36.754839+0000 mon.smithi156 (mon.1) 1 : cluster [INF] mon.smithi156 calling monitor election 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:37.166743+0000 mon.smithi006 (mon.0) 204 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi156.fphsnp/crt"}]: dispatch 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:37.752734+0000 mon.smithi006 (mon.0) 205 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:38.490173+0000 mgr.smithi006.odexej (mgr.14176) 52 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:38.752966+0000 mon.smithi006 (mon.0) 206 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:39.752933+0000 mon.smithi006 (mon.0) 207 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.784369+0000 mon.smithi006 (mon.0) 208 : cluster [INF] mon.smithi006 is new leader, mons smithi006,smithi156 in quorum (ranks 0,1) 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.791285+0000 mon.smithi006 (mon.0) 209 : cluster [DBG] monmap e2: 2 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],smithi156=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.792380+0000 mon.smithi006 (mon.0) 210 : cluster [DBG] fsmap 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.792476+0000 mon.smithi006 (mon.0) 211 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.792852+0000 mon.smithi006 (mon.0) 212 : cluster [DBG] mgrmap e17: smithi006.odexej(active, since 65s) 2023-11-17T05:34:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.794706+0000 mon.smithi006 (mon.0) 213 : cluster [DBG] Standby manager daemon smithi156.fphsnp started 2023-11-17T05:34:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:39.797836+0000 mon.smithi006 (mon.0) 214 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-17T05:34:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.801319+0000 mon.smithi006 (mon.0) 215 : cluster [INF] overall HEALTH_OK 2023-11-17T05:34:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:39.802257+0000 mon.smithi006 (mon.0) 216 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi156.fphsnp/key"}]: dispatch 2023-11-17T05:34:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:39.803241+0000 mon.smithi006 (mon.0) 217 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-17T05:34:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:39 smithi006 bash[26022]: audit 2023-11-17T05:34:39.807961+0000 mon.smithi006 (mon.0) 218 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:40.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:34.489416+0000 mgr.smithi006.odexej (mgr.14176) 50 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:34.758701+0000 mon.smithi006 (mon.0) 199 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:34.759071+0000 mon.smithi006 (mon.0) 200 : cluster [INF] mon.smithi006 calling monitor election 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:34.780822+0000 mon.smithi006 (mon.0) 201 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:35.752424+0000 mon.smithi006 (mon.0) 202 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:36.489798+0000 mgr.smithi006.odexej (mgr.14176) 51 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:36.752432+0000 mon.smithi006 (mon.0) 203 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:36.754839+0000 mon.smithi156 (mon.1) 1 : cluster [INF] mon.smithi156 calling monitor election 2023-11-17T05:34:40.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:37.166743+0000 mon.smithi006 (mon.0) 204 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi156.fphsnp/crt"}]: dispatch 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:37.752734+0000 mon.smithi006 (mon.0) 205 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:38.490173+0000 mgr.smithi006.odexej (mgr.14176) 52 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:38.752966+0000 mon.smithi006 (mon.0) 206 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:39.752933+0000 mon.smithi006 (mon.0) 207 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.784369+0000 mon.smithi006 (mon.0) 208 : cluster [INF] mon.smithi006 is new leader, mons smithi006,smithi156 in quorum (ranks 0,1) 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.791285+0000 mon.smithi006 (mon.0) 209 : cluster [DBG] monmap e2: 2 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],smithi156=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.792380+0000 mon.smithi006 (mon.0) 210 : cluster [DBG] fsmap 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.792476+0000 mon.smithi006 (mon.0) 211 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.792852+0000 mon.smithi006 (mon.0) 212 : cluster [DBG] mgrmap e17: smithi006.odexej(active, since 65s) 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.794706+0000 mon.smithi006 (mon.0) 213 : cluster [DBG] Standby manager daemon smithi156.fphsnp started 2023-11-17T05:34:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:39.797836+0000 mon.smithi006 (mon.0) 214 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-17T05:34:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.801319+0000 mon.smithi006 (mon.0) 215 : cluster [INF] overall HEALTH_OK 2023-11-17T05:34:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:39.802257+0000 mon.smithi006 (mon.0) 216 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi156.fphsnp/key"}]: dispatch 2023-11-17T05:34:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:39.803241+0000 mon.smithi006 (mon.0) 217 : audit [DBG] from='mgr.? 172.21.15.156:0/117460982' entity='mgr.smithi156.fphsnp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-17T05:34:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:39 smithi156 bash[29655]: audit 2023-11-17T05:34:39.807961+0000 mon.smithi006 (mon.0) 218 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:40 smithi006 bash[26022]: cluster 2023-11-17T05:34:39.861757+0000 mon.smithi006 (mon.0) 219 : cluster [DBG] mgrmap e18: smithi006.odexej(active, since 65s), standbys: smithi156.fphsnp 2023-11-17T05:34:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:40 smithi006 bash[26022]: audit 2023-11-17T05:34:39.863422+0000 mon.smithi006 (mon.0) 220 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi156.fphsnp", "id": "smithi156.fphsnp"}]: dispatch 2023-11-17T05:34:41.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:40 smithi006 bash[26022]: audit 2023-11-17T05:34:40.753297+0000 mon.smithi006 (mon.0) 221 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:41.176 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-17T05:34:41.176 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:34:34.753211Z","created":"2023-11-17T05:31:31.143165Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi156","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-17T05:34:41.179 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2023-11-17T05:34:41.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:40 smithi156 bash[29655]: cluster 2023-11-17T05:34:39.861757+0000 mon.smithi006 (mon.0) 219 : cluster [DBG] mgrmap e18: smithi006.odexej(active, since 65s), standbys: smithi156.fphsnp 2023-11-17T05:34:41.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:40 smithi156 bash[29655]: audit 2023-11-17T05:34:39.863422+0000 mon.smithi006 (mon.0) 220 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr metadata", "who": "smithi156.fphsnp", "id": "smithi156.fphsnp"}]: dispatch 2023-11-17T05:34:41.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:40 smithi156 bash[29655]: audit 2023-11-17T05:34:40.753297+0000 mon.smithi006 (mon.0) 221 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mon metadata", "id": "smithi156"}]: dispatch 2023-11-17T05:34:41.843 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-17T05:34:41.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: cluster 2023-11-17T05:34:40.490613+0000 mgr.smithi006.odexej (mgr.14176) 53 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.074705+0000 mon.smithi006 (mon.0) 222 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.076673+0000 mon.smithi006 (mon.0) 223 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.078395+0000 mon.smithi006 (mon.0) 224 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: cephadm 2023-11-17T05:34:41.080768+0000 mgr.smithi006.odexej (mgr.14176) 54 : cephadm [INF] Updating smithi006:/etc/ceph/ceph.conf 2023-11-17T05:34:42.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: cephadm 2023-11-17T05:34:41.081390+0000 mgr.smithi006.odexej (mgr.14176) 55 : cephadm [INF] Updating smithi156:/etc/ceph/ceph.conf 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.106124+0000 mon.smithi006 (mon.0) 225 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.112563+0000 mon.smithi006 (mon.0) 226 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.127418+0000 mon.smithi006 (mon.0) 227 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: cephadm 2023-11-17T05:34:41.134597+0000 mgr.smithi006.odexej (mgr.14176) 56 : cephadm [INF] Reconfiguring mon.smithi006 (monmap changed)... 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.135190+0000 mon.smithi006 (mon.0) 228 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.136644+0000 mon.smithi006 (mon.0) 229 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.138084+0000 mon.smithi006 (mon.0) 230 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: cephadm 2023-11-17T05:34:41.139385+0000 mgr.smithi006.odexej (mgr.14176) 57 : cephadm [INF] Reconfiguring daemon mon.smithi006 on smithi006 2023-11-17T05:34:42.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:41 smithi006 bash[26022]: audit 2023-11-17T05:34:41.174174+0000 mon.smithi006 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.156:0/1109843647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:42.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: cluster 2023-11-17T05:34:40.490613+0000 mgr.smithi006.odexej (mgr.14176) 53 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:42.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.074705+0000 mon.smithi006 (mon.0) 222 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.076673+0000 mon.smithi006 (mon.0) 223 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:42.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.078395+0000 mon.smithi006 (mon.0) 224 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: cephadm 2023-11-17T05:34:41.080768+0000 mgr.smithi006.odexej (mgr.14176) 54 : cephadm [INF] Updating smithi006:/etc/ceph/ceph.conf 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: cephadm 2023-11-17T05:34:41.081390+0000 mgr.smithi006.odexej (mgr.14176) 55 : cephadm [INF] Updating smithi156:/etc/ceph/ceph.conf 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.106124+0000 mon.smithi006 (mon.0) 225 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.112563+0000 mon.smithi006 (mon.0) 226 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.127418+0000 mon.smithi006 (mon.0) 227 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: cephadm 2023-11-17T05:34:41.134597+0000 mgr.smithi006.odexej (mgr.14176) 56 : cephadm [INF] Reconfiguring mon.smithi006 (monmap changed)... 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.135190+0000 mon.smithi006 (mon.0) 228 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.136644+0000 mon.smithi006 (mon.0) 229 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.138084+0000 mon.smithi006 (mon.0) 230 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: cephadm 2023-11-17T05:34:41.139385+0000 mgr.smithi006.odexej (mgr.14176) 57 : cephadm [INF] Reconfiguring daemon mon.smithi006 on smithi006 2023-11-17T05:34:42.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:41 smithi156 bash[29655]: audit 2023-11-17T05:34:41.174174+0000 mon.smithi006 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.156:0/1109843647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:42.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:42 smithi006 bash[26022]: cluster 2023-11-17T05:34:42.491142+0000 mgr.smithi006.odexej (mgr.14176) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:43.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:42 smithi156 bash[29655]: cluster 2023-11-17T05:34:42.491142+0000 mgr.smithi006.odexej (mgr.14176) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: audit 2023-11-17T05:34:43.632954+0000 mon.smithi006 (mon.0) 232 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: cephadm 2023-11-17T05:34:43.634260+0000 mgr.smithi006.odexej (mgr.14176) 59 : cephadm [INF] Reconfiguring mgr.smithi006.odexej (monmap changed)... 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: audit 2023-11-17T05:34:43.634815+0000 mon.smithi006 (mon.0) 233 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi006.odexej", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: audit 2023-11-17T05:34:43.636650+0000 mon.smithi006 (mon.0) 234 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: audit 2023-11-17T05:34:43.638224+0000 mon.smithi006 (mon.0) 235 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:44.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:44 smithi006 bash[26022]: cephadm 2023-11-17T05:34:43.639431+0000 mgr.smithi006.odexej (mgr.14176) 60 : cephadm [INF] Reconfiguring daemon mgr.smithi006.odexej on smithi006 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: audit 2023-11-17T05:34:43.632954+0000 mon.smithi006 (mon.0) 232 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: cephadm 2023-11-17T05:34:43.634260+0000 mgr.smithi006.odexej (mgr.14176) 59 : cephadm [INF] Reconfiguring mgr.smithi006.odexej (monmap changed)... 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: audit 2023-11-17T05:34:43.634815+0000 mon.smithi006 (mon.0) 233 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi006.odexej", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: audit 2023-11-17T05:34:43.636650+0000 mon.smithi006 (mon.0) 234 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: audit 2023-11-17T05:34:43.638224+0000 mon.smithi006 (mon.0) 235 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:44.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:44 smithi156 bash[29655]: cephadm 2023-11-17T05:34:43.639431+0000 mgr.smithi006.odexej (mgr.14176) 60 : cephadm [INF] Reconfiguring daemon mgr.smithi006.odexej on smithi006 2023-11-17T05:34:45.725 INFO:teuthology.orchestra.run.smithi006.stdout:# minimal ceph.conf for 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:34:45.726 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2023-11-17T05:34:45.726 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 7d14d910-850a-11ee-8dbe-212e2dc638e7 2023-11-17T05:34:45.726 INFO:teuthology.orchestra.run.smithi006.stdout: mon_host = [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-11-17T05:34:46.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:45 smithi006 bash[26022]: cluster 2023-11-17T05:34:44.491632+0000 mgr.smithi006.odexej (mgr.14176) 61 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:46.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:45 smithi156 bash[29655]: cluster 2023-11-17T05:34:44.491632+0000 mgr.smithi006.odexej (mgr.14176) 61 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:46.463 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-17T05:34:46.463 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:34:46.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2023-11-17T05:34:46.478 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:34:46.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:34:46.536 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:34:46.536 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2023-11-17T05:34:46.551 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:34:46.552 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:34:46.608 INFO:tasks.cephadm:Deploying OSDs... 2023-11-17T05:34:46.608 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:34:46.608 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:34:46.614 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T05:34:46.615 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme1n1 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme1n1 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 473 Links: 1 Device type: 103,2 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:32:57.052352841 +0000 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.663 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:34:46.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-11-17T05:34:46.714 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:46 smithi006 bash[26022]: audit 2023-11-17T05:34:45.724132+0000 mon.smithi006 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.6:0/867356999' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:46.714 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:46 smithi006 bash[26022]: audit 2023-11-17T05:34:45.912556+0000 mon.smithi006 (mon.0) 237 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:46.716 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:34:46.716 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:34:46.716 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000479053 s, 1.1 MB/s 2023-11-17T05:34:46.717 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-11-17T05:34:46.769 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme2n1 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme2n1 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 479 Links: 1 Device type: 103,4 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:32:57.068352460 +0000 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.819 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:34:46.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-11-17T05:34:46.878 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:34:46.878 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:34:46.878 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000345691 s, 1.5 MB/s 2023-11-17T05:34:46.879 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-11-17T05:34:46.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:46 smithi156 bash[29655]: audit 2023-11-17T05:34:45.724132+0000 mon.smithi006 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.6:0/867356999' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:46.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:46 smithi156 bash[29655]: audit 2023-11-17T05:34:45.912556+0000 mon.smithi006 (mon.0) 237 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:46.932 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme3n1 2023-11-17T05:34:46.982 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme3n1 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 485 Links: 1 Device type: 103,6 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:32:57.080352175 +0000 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:32:57.020353601 +0000 2023-11-17T05:34:46.983 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:34:46.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-11-17T05:34:47.042 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:34:47.042 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:34:47.042 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000456984 s, 1.1 MB/s 2023-11-17T05:34:47.043 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-11-17T05:34:47.092 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme4n1 2023-11-17T05:34:47.147 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme4n1 2023-11-17T05:34:47.147 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:47.147 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 491 Links: 1 Device type: 103,8 2023-11-17T05:34:47.148 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:47.148 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-11-17 05:32:57.092351889 +0000 2023-11-17T05:34:47.148 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-11-17 05:32:57.024353507 +0000 2023-11-17T05:34:47.148 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-11-17 05:32:57.024353507 +0000 2023-11-17T05:34:47.148 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-11-17T05:34:47.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-11-17T05:34:47.200 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-11-17T05:34:47.200 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-11-17T05:34:47.200 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000507448 s, 1.0 MB/s 2023-11-17T05:34:47.201 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-11-17T05:34:47.256 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:34:47.256 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T05:34:47.263 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T05:34:47.263 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/nvme1n1 2023-11-17T05:34:47.312 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/nvme1n1 2023-11-17T05:34:47.312 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:47.312 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 473 Links: 1 Device type: 103,2 2023-11-17T05:34:47.312 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:47.313 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:34:20.918906575 +0000 2023-11-17T05:34:47.313 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:34:20.874905818 +0000 2023-11-17T05:34:47.313 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:34:20.874905818 +0000 2023-11-17T05:34:47.313 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:34:47.313 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-11-17T05:34:47.367 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:34:47.367 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:34:47.367 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000353488 s, 1.4 MB/s 2023-11-17T05:34:47.368 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-11-17T05:34:47.421 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/nvme2n1 2023-11-17T05:34:47.473 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/nvme2n1 2023-11-17T05:34:47.473 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:47.473 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 479 Links: 1 Device type: 103,4 2023-11-17T05:34:47.474 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:47.474 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:34:20.926906713 +0000 2023-11-17T05:34:47.474 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.474 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.474 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:34:47.474 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-11-17T05:34:47.531 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:34:47.531 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:34:47.531 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000342466 s, 1.5 MB/s 2023-11-17T05:34:47.532 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-11-17T05:34:47.581 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/nvme3n1 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/nvme3n1 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 485 Links: 1 Device type: 103,6 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:34:20.938906919 +0000 2023-11-17T05:34:47.632 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.633 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.633 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:34:47.633 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-11-17T05:34:47.688 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:34:47.688 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:34:47.688 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000260562 s, 2.0 MB/s 2023-11-17T05:34:47.689 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-11-17T05:34:47.738 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/nvme4n1 2023-11-17T05:34:47.788 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/nvme4n1 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 491 Links: 1 Device type: 103,8 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-17 05:34:20.946907057 +0000 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-17 05:34:20.878905888 +0000 2023-11-17T05:34:47.789 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-17T05:34:47.789 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-11-17T05:34:47.843 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-17T05:34:47.843 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-17T05:34:47.843 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000280845 s, 1.8 MB/s 2023-11-17T05:34:47.844 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-11-17T05:34:47.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch apply osd --all-available-devices 2023-11-17T05:34:47.952 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:47 smithi156 bash[29655]: cephadm 2023-11-17T05:34:45.913848+0000 mgr.smithi006.odexej (mgr.14176) 62 : cephadm [INF] Reconfiguring alertmanager.smithi006 (dependencies changed)... 2023-11-17T05:34:47.952 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:47 smithi156 bash[29655]: cephadm 2023-11-17T05:34:45.920887+0000 mgr.smithi006.odexej (mgr.14176) 63 : cephadm [INF] Reconfiguring daemon alertmanager.smithi006 on smithi006 2023-11-17T05:34:47.953 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:47 smithi156 bash[29655]: cluster 2023-11-17T05:34:46.492088+0000 mgr.smithi006.odexej (mgr.14176) 64 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:48.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:47 smithi006 bash[26022]: cephadm 2023-11-17T05:34:45.913848+0000 mgr.smithi006.odexej (mgr.14176) 62 : cephadm [INF] Reconfiguring alertmanager.smithi006 (dependencies changed)... 2023-11-17T05:34:48.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:47 smithi006 bash[26022]: cephadm 2023-11-17T05:34:45.920887+0000 mgr.smithi006.odexej (mgr.14176) 63 : cephadm [INF] Reconfiguring daemon alertmanager.smithi006 on smithi006 2023-11-17T05:34:48.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:47 smithi006 bash[26022]: cluster 2023-11-17T05:34:46.492088+0000 mgr.smithi006.odexej (mgr.14176) 64 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:49.992 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:49 smithi006 bash[26022]: cluster 2023-11-17T05:34:48.492572+0000 mgr.smithi006.odexej (mgr.14176) 65 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:50.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:49 smithi156 bash[29655]: cluster 2023-11-17T05:34:48.492572+0000 mgr.smithi006.odexej (mgr.14176) 65 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:51.010 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled osd.all-available-devices update... 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: audit 2023-11-17T05:34:50.111130+0000 mon.smithi006 (mon.0) 238 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: cephadm 2023-11-17T05:34:50.112111+0000 mgr.smithi006.odexej (mgr.14176) 66 : cephadm [INF] Reconfiguring crash.smithi006 (monmap changed)... 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: audit 2023-11-17T05:34:50.112613+0000 mon.smithi006 (mon.0) 239 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: audit 2023-11-17T05:34:50.114062+0000 mon.smithi006 (mon.0) 240 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: cephadm 2023-11-17T05:34:50.115099+0000 mgr.smithi006.odexej (mgr.14176) 67 : cephadm [INF] Reconfiguring daemon crash.smithi006 on smithi006 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: cluster 2023-11-17T05:34:50.493156+0000 mgr.smithi006.odexej (mgr.14176) 68 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:51.370 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:51 smithi006 bash[26022]: audit 2023-11-17T05:34:51.007433+0000 mon.smithi006 (mon.0) 241 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: audit 2023-11-17T05:34:50.111130+0000 mon.smithi006 (mon.0) 238 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: cephadm 2023-11-17T05:34:50.112111+0000 mgr.smithi006.odexej (mgr.14176) 66 : cephadm [INF] Reconfiguring crash.smithi006 (monmap changed)... 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: audit 2023-11-17T05:34:50.112613+0000 mon.smithi006 (mon.0) 239 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: audit 2023-11-17T05:34:50.114062+0000 mon.smithi006 (mon.0) 240 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: cephadm 2023-11-17T05:34:50.115099+0000 mgr.smithi006.odexej (mgr.14176) 67 : cephadm [INF] Reconfiguring daemon crash.smithi006 on smithi006 2023-11-17T05:34:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: cluster 2023-11-17T05:34:50.493156+0000 mgr.smithi006.odexej (mgr.14176) 68 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:51.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:51 smithi156 bash[29655]: audit 2023-11-17T05:34:51.007433+0000 mon.smithi006 (mon.0) 241 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:34:51.597 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-11-17T05:35:05.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:05.942 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:52 smithi006 bash[26022]: audit 2023-11-17T05:34:50.995713+0000 mgr.smithi006.odexej (mgr.14176) 69 : audit [DBG] from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:35:05.942 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:52 smithi006 bash[26022]: cephadm 2023-11-17T05:34:50.999630+0000 mgr.smithi006.odexej (mgr.14176) 70 : cephadm [INF] Marking host: smithi006 for OSDSpec preview refresh. 2023-11-17T05:35:05.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:52 smithi006 bash[26022]: cephadm 2023-11-17T05:34:50.999801+0000 mgr.smithi006.odexej (mgr.14176) 71 : cephadm [INF] Marking host: smithi156 for OSDSpec preview refresh. 2023-11-17T05:35:05.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:52 smithi006 bash[26022]: cephadm 2023-11-17T05:34:51.000530+0000 mgr.smithi006.odexej (mgr.14176) 72 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2023-11-17T05:35:05.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:53 smithi006 bash[26022]: audit 2023-11-17T05:34:52.252734+0000 mon.smithi006 (mon.0) 242 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:53 smithi006 bash[26022]: cephadm 2023-11-17T05:34:52.254587+0000 mgr.smithi006.odexej (mgr.14176) 73 : cephadm [INF] Reconfiguring prometheus.smithi006 (dependencies changed)... 2023-11-17T05:35:05.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:53 smithi006 bash[26022]: cephadm 2023-11-17T05:34:52.262309+0000 mgr.smithi006.odexej (mgr.14176) 74 : cephadm [INF] Reconfiguring daemon prometheus.smithi006 on smithi006 2023-11-17T05:35:05.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:53 smithi006 bash[26022]: cluster 2023-11-17T05:34:52.493614+0000 mgr.smithi006.odexej (mgr.14176) 75 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:55 smithi006 bash[26022]: cluster 2023-11-17T05:34:54.494057+0000 mgr.smithi006.odexej (mgr.14176) 76 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: audit 2023-11-17T05:34:56.455272+0000 mon.smithi006 (mon.0) 243 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: cephadm 2023-11-17T05:34:56.456358+0000 mgr.smithi006.odexej (mgr.14176) 77 : cephadm [INF] Reconfiguring crash.smithi156 (monmap changed)... 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: audit 2023-11-17T05:34:56.456891+0000 mon.smithi006 (mon.0) 244 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi156", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: audit 2023-11-17T05:34:56.458512+0000 mon.smithi006 (mon.0) 245 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: cephadm 2023-11-17T05:34:56.459813+0000 mgr.smithi006.odexej (mgr.14176) 78 : cephadm [INF] Reconfiguring daemon crash.smithi156 on smithi156 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:57 smithi006 bash[26022]: cluster 2023-11-17T05:34:56.494508+0000 mgr.smithi006.odexej (mgr.14176) 79 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: audit 2023-11-17T05:34:58.292434+0000 mon.smithi006 (mon.0) 246 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: cephadm 2023-11-17T05:34:58.294006+0000 mgr.smithi006.odexej (mgr.14176) 80 : cephadm [INF] Reconfiguring mgr.smithi156.fphsnp (monmap changed)... 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: audit 2023-11-17T05:34:58.294689+0000 mon.smithi006 (mon.0) 247 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi156.fphsnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: audit 2023-11-17T05:34:58.296840+0000 mon.smithi006 (mon.0) 248 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: audit 2023-11-17T05:34:58.298573+0000 mon.smithi006 (mon.0) 249 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: cephadm 2023-11-17T05:34:58.300211+0000 mgr.smithi006.odexej (mgr.14176) 81 : cephadm [INF] Reconfiguring daemon mgr.smithi156.fphsnp on smithi156 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:34:59 smithi006 bash[26022]: cluster 2023-11-17T05:34:58.494896+0000 mgr.smithi006.odexej (mgr.14176) 82 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: audit 2023-11-17T05:34:59.940226+0000 mon.smithi006 (mon.0) 250 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: cephadm 2023-11-17T05:34:59.941502+0000 mgr.smithi006.odexej (mgr.14176) 83 : cephadm [INF] Reconfiguring mon.smithi156 (monmap changed)... 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: audit 2023-11-17T05:34:59.942071+0000 mon.smithi006 (mon.0) 251 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: audit 2023-11-17T05:34:59.943870+0000 mon.smithi006 (mon.0) 252 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: audit 2023-11-17T05:34:59.945549+0000 mon.smithi006 (mon.0) 253 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: cephadm 2023-11-17T05:34:59.946947+0000 mgr.smithi006.odexej (mgr.14176) 84 : cephadm [INF] Reconfiguring daemon mon.smithi156 on smithi156 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:00 smithi006 bash[26022]: cluster 2023-11-17T05:35:00.495341+0000 mgr.smithi006.odexej (mgr.14176) 85 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.563488+0000 mon.smithi006 (mon.0) 254 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.571568+0000 mon.smithi006 (mon.0) 255 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.572562+0000 mgr.smithi006.odexej (mgr.14176) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.580908+0000 mon.smithi006 (mon.0) 256 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.581951+0000 mgr.smithi006.odexej (mgr.14176) 87 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:02 smithi006 bash[26022]: audit 2023-11-17T05:35:01.585319+0000 mon.smithi006 (mon.0) 257 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:03 smithi006 bash[26022]: cluster 2023-11-17T05:35:02.495780+0000 mgr.smithi006.odexej (mgr.14176) 88 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:05 smithi006 bash[26022]: cluster 2023-11-17T05:35:04.496216+0000 mgr.smithi006.odexej (mgr.14176) 89 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:52 smithi156 bash[29655]: audit 2023-11-17T05:34:50.995713+0000 mgr.smithi006.odexej (mgr.14176) 69 : audit [DBG] from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:52 smithi156 bash[29655]: cephadm 2023-11-17T05:34:50.999630+0000 mgr.smithi006.odexej (mgr.14176) 70 : cephadm [INF] Marking host: smithi006 for OSDSpec preview refresh. 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:52 smithi156 bash[29655]: cephadm 2023-11-17T05:34:50.999801+0000 mgr.smithi006.odexej (mgr.14176) 71 : cephadm [INF] Marking host: smithi156 for OSDSpec preview refresh. 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:52 smithi156 bash[29655]: cephadm 2023-11-17T05:34:51.000530+0000 mgr.smithi006.odexej (mgr.14176) 72 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:53 smithi156 bash[29655]: audit 2023-11-17T05:34:52.252734+0000 mon.smithi006 (mon.0) 242 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:53 smithi156 bash[29655]: cephadm 2023-11-17T05:34:52.254587+0000 mgr.smithi006.odexej (mgr.14176) 73 : cephadm [INF] Reconfiguring prometheus.smithi006 (dependencies changed)... 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:53 smithi156 bash[29655]: cephadm 2023-11-17T05:34:52.262309+0000 mgr.smithi006.odexej (mgr.14176) 74 : cephadm [INF] Reconfiguring daemon prometheus.smithi006 on smithi006 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:53 smithi156 bash[29655]: cluster 2023-11-17T05:34:52.493614+0000 mgr.smithi006.odexej (mgr.14176) 75 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:55 smithi156 bash[29655]: cluster 2023-11-17T05:34:54.494057+0000 mgr.smithi006.odexej (mgr.14176) 76 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: audit 2023-11-17T05:34:56.455272+0000 mon.smithi006 (mon.0) 243 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: cephadm 2023-11-17T05:34:56.456358+0000 mgr.smithi006.odexej (mgr.14176) 77 : cephadm [INF] Reconfiguring crash.smithi156 (monmap changed)... 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: audit 2023-11-17T05:34:56.456891+0000 mon.smithi006 (mon.0) 244 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi156", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: audit 2023-11-17T05:34:56.458512+0000 mon.smithi006 (mon.0) 245 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: cephadm 2023-11-17T05:34:56.459813+0000 mgr.smithi006.odexej (mgr.14176) 78 : cephadm [INF] Reconfiguring daemon crash.smithi156 on smithi156 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:57 smithi156 bash[29655]: cluster 2023-11-17T05:34:56.494508+0000 mgr.smithi006.odexej (mgr.14176) 79 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: audit 2023-11-17T05:34:58.292434+0000 mon.smithi006 (mon.0) 246 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: cephadm 2023-11-17T05:34:58.294006+0000 mgr.smithi006.odexej (mgr.14176) 80 : cephadm [INF] Reconfiguring mgr.smithi156.fphsnp (monmap changed)... 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: audit 2023-11-17T05:34:58.294689+0000 mon.smithi006 (mon.0) 247 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi156.fphsnp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: audit 2023-11-17T05:34:58.296840+0000 mon.smithi006 (mon.0) 248 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: audit 2023-11-17T05:34:58.298573+0000 mon.smithi006 (mon.0) 249 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: cephadm 2023-11-17T05:34:58.300211+0000 mgr.smithi006.odexej (mgr.14176) 81 : cephadm [INF] Reconfiguring daemon mgr.smithi156.fphsnp on smithi156 2023-11-17T05:35:05.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:34:59 smithi156 bash[29655]: cluster 2023-11-17T05:34:58.494896+0000 mgr.smithi006.odexej (mgr.14176) 82 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: audit 2023-11-17T05:34:59.940226+0000 mon.smithi006 (mon.0) 250 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: cephadm 2023-11-17T05:34:59.941502+0000 mgr.smithi006.odexej (mgr.14176) 83 : cephadm [INF] Reconfiguring mon.smithi156 (monmap changed)... 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: audit 2023-11-17T05:34:59.942071+0000 mon.smithi006 (mon.0) 251 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: audit 2023-11-17T05:34:59.943870+0000 mon.smithi006 (mon.0) 252 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: audit 2023-11-17T05:34:59.945549+0000 mon.smithi006 (mon.0) 253 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: cephadm 2023-11-17T05:34:59.946947+0000 mgr.smithi006.odexej (mgr.14176) 84 : cephadm [INF] Reconfiguring daemon mon.smithi156 on smithi156 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:00 smithi156 bash[29655]: cluster 2023-11-17T05:35:00.495341+0000 mgr.smithi006.odexej (mgr.14176) 85 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.563488+0000 mon.smithi006 (mon.0) 254 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.571568+0000 mon.smithi006 (mon.0) 255 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.572562+0000 mgr.smithi006.odexej (mgr.14176) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.580908+0000 mon.smithi006 (mon.0) 256 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.581951+0000 mgr.smithi006.odexej (mgr.14176) 87 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:02 smithi156 bash[29655]: audit 2023-11-17T05:35:01.585319+0000 mon.smithi006 (mon.0) 257 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:35:05.993 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:03 smithi156 bash[29655]: cluster 2023-11-17T05:35:02.495780+0000 mgr.smithi006.odexej (mgr.14176) 88 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.994 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:05 smithi156 bash[29655]: cluster 2023-11-17T05:35:04.496216+0000 mgr.smithi006.odexej (mgr.14176) 89 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:07.360 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:07 smithi006 bash[26022]: cluster 2023-11-17T05:35:06.496664+0000 mgr.smithi006.odexej (mgr.14176) 90 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:07.360 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:07 smithi006 bash[26022]: audit 2023-11-17T05:35:06.979419+0000 mon.smithi006 (mon.0) 258 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:07.360 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:07 smithi006 bash[26022]: audit 2023-11-17T05:35:06.986664+0000 mon.smithi006 (mon.0) 259 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:07.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:07 smithi156 bash[29655]: cluster 2023-11-17T05:35:06.496664+0000 mgr.smithi006.odexej (mgr.14176) 90 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:07.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:07 smithi156 bash[29655]: audit 2023-11-17T05:35:06.979419+0000 mon.smithi006 (mon.0) 258 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:07.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:07 smithi156 bash[29655]: audit 2023-11-17T05:35:06.986664+0000 mon.smithi006 (mon.0) 259 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:08.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.610241+0000 mon.smithi006 (mon.0) 260 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:08.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.904871+0000 mon.smithi006 (mon.0) 261 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:09.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.906822+0000 mon.smithi006 (mon.0) 262 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.908013+0000 mon.smithi006 (mon.0) 263 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.917323+0000 mon.smithi006 (mon.0) 264 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.925778+0000 mon.smithi006 (mon.0) 265 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.931188+0000 mon.smithi006 (mon.0) 266 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.933581+0000 mon.smithi006 (mon.0) 267 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.933830+0000 mon.smithi006 (mon.0) 268 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:08 smithi006 bash[26022]: audit 2023-11-17T05:35:07.934571+0000 mon.smithi006 (mon.0) 269 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.610241+0000 mon.smithi006 (mon.0) 260 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.904871+0000 mon.smithi006 (mon.0) 261 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.906822+0000 mon.smithi006 (mon.0) 262 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.908013+0000 mon.smithi006 (mon.0) 263 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.917323+0000 mon.smithi006 (mon.0) 264 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.925778+0000 mon.smithi006 (mon.0) 265 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.931188+0000 mon.smithi006 (mon.0) 266 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.933581+0000 mon.smithi006 (mon.0) 267 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.933830+0000 mon.smithi006 (mon.0) 268 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:08 smithi156 bash[29655]: audit 2023-11-17T05:35:07.934571+0000 mon.smithi006 (mon.0) 269 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:35:09.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:09.604 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2023-11-17T05:35:09.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:09 smithi006 bash[26022]: cluster 2023-11-17T05:35:08.497131+0000 mgr.smithi006.odexej (mgr.14176) 91 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:09.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:09 smithi006 bash[26022]: audit 2023-11-17T05:35:09.042871+0000 mon.smithi006 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.6:0/3701051700' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:09.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:09 smithi156 bash[29655]: cluster 2023-11-17T05:35:08.497131+0000 mgr.smithi006.odexej (mgr.14176) 91 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:09.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:09 smithi156 bash[29655]: audit 2023-11-17T05:35:09.042871+0000 mon.smithi006 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.6:0/3701051700' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:10.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:11.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:11 smithi006 bash[26022]: cluster 2023-11-17T05:35:10.497592+0000 mgr.smithi006.odexej (mgr.14176) 92 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:11.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:11 smithi156 bash[29655]: cluster 2023-11-17T05:35:10.497592+0000 mgr.smithi006.odexej (mgr.14176) 92 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:13.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:13.751 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:13 smithi006 bash[26022]: cluster 2023-11-17T05:35:12.498149+0000 mgr.smithi006.odexej (mgr.14176) 93 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:13.857 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:13 smithi156 bash[29655]: cluster 2023-11-17T05:35:12.498149+0000 mgr.smithi006.odexej (mgr.14176) 93 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:14.456 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2023-11-17T05:35:14.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:14 smithi006 bash[26022]: audit 2023-11-17T05:35:13.687466+0000 mon.smithi006 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.6:0/1655625405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:14.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:14 smithi156 bash[29655]: audit 2023-11-17T05:35:13.687466+0000 mon.smithi006 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.6:0/1655625405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:15.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:15.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:15 smithi006 bash[26022]: cluster 2023-11-17T05:35:14.498652+0000 mgr.smithi006.odexej (mgr.14176) 94 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:15.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:15 smithi156 bash[29655]: cluster 2023-11-17T05:35:14.498652+0000 mgr.smithi006.odexej (mgr.14176) 94 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:17.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: audit 2023-11-17T05:35:16.346975+0000 mon.smithi156 (mon.1) 2 : audit [INF] from='client.? 172.21.15.156:0/634875929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]: dispatch 2023-11-17T05:35:17.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: audit 2023-11-17T05:35:16.348039+0000 mon.smithi006 (mon.0) 272 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]: dispatch 2023-11-17T05:35:17.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: audit 2023-11-17T05:35:16.355748+0000 mon.smithi006 (mon.0) 273 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]': finished 2023-11-17T05:35:17.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: cluster 2023-11-17T05:35:16.355950+0000 mon.smithi006 (mon.0) 274 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-11-17T05:35:17.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: audit 2023-11-17T05:35:16.356222+0000 mon.smithi006 (mon.0) 275 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:17.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:17 smithi006 bash[26022]: cluster 2023-11-17T05:35:16.499097+0000 mgr.smithi006.odexej (mgr.14176) 95 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: audit 2023-11-17T05:35:16.346975+0000 mon.smithi156 (mon.1) 2 : audit [INF] from='client.? 172.21.15.156:0/634875929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]: dispatch 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: audit 2023-11-17T05:35:16.348039+0000 mon.smithi006 (mon.0) 272 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]: dispatch 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: audit 2023-11-17T05:35:16.355748+0000 mon.smithi006 (mon.0) 273 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd001464-a43a-4e8e-b89f-737e2fb6afcc"}]': finished 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: cluster 2023-11-17T05:35:16.355950+0000 mon.smithi006 (mon.0) 274 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: audit 2023-11-17T05:35:16.356222+0000 mon.smithi006 (mon.0) 275 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:17 smithi156 bash[29655]: cluster 2023-11-17T05:35:16.499097+0000 mgr.smithi006.odexej (mgr.14176) 95 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:18.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: audit 2023-11-17T05:35:17.384869+0000 mon.smithi156 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.156:0/1392059215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:18.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: audit 2023-11-17T05:35:18.023286+0000 mon.smithi006 (mon.0) 276 : audit [INF] from='client.? 172.21.15.6:0/1694520230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eefe282d-2c80-4ba7-9d5b-5352b05ae877"}]: dispatch 2023-11-17T05:35:18.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: audit 2023-11-17T05:35:18.030980+0000 mon.smithi006 (mon.0) 277 : audit [INF] from='client.? 172.21.15.6:0/1694520230' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eefe282d-2c80-4ba7-9d5b-5352b05ae877"}]': finished 2023-11-17T05:35:18.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: cluster 2023-11-17T05:35:18.031135+0000 mon.smithi006 (mon.0) 278 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2023-11-17T05:35:18.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: audit 2023-11-17T05:35:18.031385+0000 mon.smithi006 (mon.0) 279 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:18.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:18 smithi006 bash[26022]: audit 2023-11-17T05:35:18.032516+0000 mon.smithi006 (mon.0) 280 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: audit 2023-11-17T05:35:17.384869+0000 mon.smithi156 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.156:0/1392059215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: audit 2023-11-17T05:35:18.023286+0000 mon.smithi006 (mon.0) 276 : audit [INF] from='client.? 172.21.15.6:0/1694520230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eefe282d-2c80-4ba7-9d5b-5352b05ae877"}]: dispatch 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: audit 2023-11-17T05:35:18.030980+0000 mon.smithi006 (mon.0) 277 : audit [INF] from='client.? 172.21.15.6:0/1694520230' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eefe282d-2c80-4ba7-9d5b-5352b05ae877"}]': finished 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: cluster 2023-11-17T05:35:18.031135+0000 mon.smithi006 (mon.0) 278 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: audit 2023-11-17T05:35:18.031385+0000 mon.smithi006 (mon.0) 279 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:18 smithi156 bash[29655]: audit 2023-11-17T05:35:18.032516+0000 mon.smithi006 (mon.0) 280 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:18.910 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:19.499 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1700199318,"num_remapped_pgs":0} 2023-11-17T05:35:19.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:19 smithi006 bash[26022]: cluster 2023-11-17T05:35:18.499612+0000 mgr.smithi006.odexej (mgr.14176) 96 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:19.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:19 smithi006 bash[26022]: audit 2023-11-17T05:35:18.908989+0000 mon.smithi006 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.6:0/155823636' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:19.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:19 smithi006 bash[26022]: audit 2023-11-17T05:35:19.248743+0000 mon.smithi006 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.6:0/3514720160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:19.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:19 smithi156 bash[29655]: cluster 2023-11-17T05:35:18.499612+0000 mgr.smithi006.odexej (mgr.14176) 96 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:19.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:19 smithi156 bash[29655]: audit 2023-11-17T05:35:18.908989+0000 mon.smithi006 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.6:0/155823636' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:19.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:19 smithi156 bash[29655]: audit 2023-11-17T05:35:19.248743+0000 mon.smithi006 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.6:0/3514720160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:20.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:20.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.120257+0000 mon.smithi156 (mon.1) 4 : audit [INF] from='client.? 172.21.15.156:0/2310943473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]: dispatch 2023-11-17T05:35:20.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.121478+0000 mon.smithi006 (mon.0) 283 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]: dispatch 2023-11-17T05:35:20.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.131280+0000 mon.smithi006 (mon.0) 284 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]': finished 2023-11-17T05:35:20.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: cluster 2023-11-17T05:35:20.131501+0000 mon.smithi006 (mon.0) 285 : cluster [DBG] osdmap e8: 3 total, 0 up, 3 in 2023-11-17T05:35:20.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.131790+0000 mon.smithi006 (mon.0) 286 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:20.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.132663+0000 mon.smithi006 (mon.0) 287 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:20.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:20 smithi006 bash[26022]: audit 2023-11-17T05:35:20.133097+0000 mon.smithi006 (mon.0) 288 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.120257+0000 mon.smithi156 (mon.1) 4 : audit [INF] from='client.? 172.21.15.156:0/2310943473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]: dispatch 2023-11-17T05:35:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.121478+0000 mon.smithi006 (mon.0) 283 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]: dispatch 2023-11-17T05:35:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.131280+0000 mon.smithi006 (mon.0) 284 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1"}]': finished 2023-11-17T05:35:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: cluster 2023-11-17T05:35:20.131501+0000 mon.smithi006 (mon.0) 285 : cluster [DBG] osdmap e8: 3 total, 0 up, 3 in 2023-11-17T05:35:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.131790+0000 mon.smithi006 (mon.0) 286 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.132663+0000 mon.smithi006 (mon.0) 287 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:20 smithi156 bash[29655]: audit 2023-11-17T05:35:20.133097+0000 mon.smithi006 (mon.0) 288 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:21.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:21 smithi006 bash[26022]: cluster 2023-11-17T05:35:20.500016+0000 mgr.smithi006.odexej (mgr.14176) 97 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:21.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:21 smithi006 bash[26022]: audit 2023-11-17T05:35:21.137960+0000 mon.smithi156 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.156:0/1854509402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:21 smithi156 bash[29655]: cluster 2023-11-17T05:35:20.500016+0000 mgr.smithi006.odexej (mgr.14176) 97 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:21 smithi156 bash[29655]: audit 2023-11-17T05:35:21.137960+0000 mon.smithi156 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.156:0/1854509402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:22.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.011233+0000 mon.smithi006 (mon.0) 289 : audit [INF] from='client.? 172.21.15.6:0/776280462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96fca6d4-495b-4520-b8be-df976c05aa30"}]: dispatch 2023-11-17T05:35:22.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.019426+0000 mon.smithi006 (mon.0) 290 : audit [INF] from='client.? 172.21.15.6:0/776280462' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96fca6d4-495b-4520-b8be-df976c05aa30"}]': finished 2023-11-17T05:35:22.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: cluster 2023-11-17T05:35:22.019687+0000 mon.smithi006 (mon.0) 291 : cluster [DBG] osdmap e9: 4 total, 0 up, 4 in 2023-11-17T05:35:22.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.019885+0000 mon.smithi006 (mon.0) 292 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:22.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.020325+0000 mon.smithi006 (mon.0) 293 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:22.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.020593+0000 mon.smithi006 (mon.0) 294 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:22.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:22 smithi006 bash[26022]: audit 2023-11-17T05:35:22.020843+0000 mon.smithi006 (mon.0) 295 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.011233+0000 mon.smithi006 (mon.0) 289 : audit [INF] from='client.? 172.21.15.6:0/776280462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96fca6d4-495b-4520-b8be-df976c05aa30"}]: dispatch 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.019426+0000 mon.smithi006 (mon.0) 290 : audit [INF] from='client.? 172.21.15.6:0/776280462' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96fca6d4-495b-4520-b8be-df976c05aa30"}]': finished 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: cluster 2023-11-17T05:35:22.019687+0000 mon.smithi006 (mon.0) 291 : cluster [DBG] osdmap e9: 4 total, 0 up, 4 in 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.019885+0000 mon.smithi006 (mon.0) 292 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.020325+0000 mon.smithi006 (mon.0) 293 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.020593+0000 mon.smithi006 (mon.0) 294 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:22 smithi156 bash[29655]: audit 2023-11-17T05:35:22.020843+0000 mon.smithi006 (mon.0) 295 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:23.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:23 smithi006 bash[26022]: cluster 2023-11-17T05:35:22.500355+0000 mgr.smithi006.odexej (mgr.14176) 98 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:23.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:23 smithi156 bash[29655]: cluster 2023-11-17T05:35:22.500355+0000 mgr.smithi006.odexej (mgr.14176) 98 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:23.918 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:24.517 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"num_osds":5,"num_up_osds":0,"osd_up_since":0,"num_in_osds":5,"osd_in_since":1700199323,"num_remapped_pgs":0} 2023-11-17T05:35:24.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.776908+0000 mon.smithi006 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.6:0/3299886099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:24.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.883297+0000 mon.smithi156 (mon.1) 6 : audit [INF] from='client.? 172.21.15.156:0/1732527805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]: dispatch 2023-11-17T05:35:24.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.884191+0000 mon.smithi006 (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]: dispatch 2023-11-17T05:35:24.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.890778+0000 mon.smithi006 (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]': finished 2023-11-17T05:35:24.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: cluster 2023-11-17T05:35:23.890859+0000 mon.smithi006 (mon.0) 299 : cluster [DBG] osdmap e10: 5 total, 0 up, 5 in 2023-11-17T05:35:24.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.890958+0000 mon.smithi006 (mon.0) 300 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:24.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.891271+0000 mon.smithi006 (mon.0) 301 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:24.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.891446+0000 mon.smithi006 (mon.0) 302 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:24.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.891664+0000 mon.smithi006 (mon.0) 303 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:24.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.891857+0000 mon.smithi006 (mon.0) 304 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:24.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:24 smithi006 bash[26022]: audit 2023-11-17T05:35:23.917219+0000 mon.smithi006 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.6:0/4087707585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.776908+0000 mon.smithi006 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.6:0/3299886099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.883297+0000 mon.smithi156 (mon.1) 6 : audit [INF] from='client.? 172.21.15.156:0/1732527805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]: dispatch 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.884191+0000 mon.smithi006 (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]: dispatch 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.890778+0000 mon.smithi006 (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5866a83c-d866-4dcf-868c-7150a44f314b"}]': finished 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: cluster 2023-11-17T05:35:23.890859+0000 mon.smithi006 (mon.0) 299 : cluster [DBG] osdmap e10: 5 total, 0 up, 5 in 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.890958+0000 mon.smithi006 (mon.0) 300 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.891271+0000 mon.smithi006 (mon.0) 301 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.891446+0000 mon.smithi006 (mon.0) 302 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.891664+0000 mon.smithi006 (mon.0) 303 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.891857+0000 mon.smithi006 (mon.0) 304 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:24 smithi156 bash[29655]: audit 2023-11-17T05:35:23.917219+0000 mon.smithi006 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.6:0/4087707585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:25.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:25.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:25 smithi006 bash[26022]: cluster 2023-11-17T05:35:24.500775+0000 mgr.smithi006.odexej (mgr.14176) 99 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:25.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:25 smithi006 bash[26022]: audit 2023-11-17T05:35:24.907502+0000 mon.smithi156 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.156:0/167564683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:25.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:25 smithi156 bash[29655]: cluster 2023-11-17T05:35:24.500775+0000 mgr.smithi006.odexej (mgr.14176) 99 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:25.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:25 smithi156 bash[29655]: audit 2023-11-17T05:35:24.907502+0000 mon.smithi156 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.156:0/167564683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:27.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: cluster 2023-11-17T05:35:26.501118+0000 mgr.smithi006.odexej (mgr.14176) 100 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.561150+0000 mon.smithi006 (mon.0) 306 : audit [INF] from='client.? 172.21.15.6:0/2410430036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e3f232f-34c0-4619-b611-c288652cf48a"}]: dispatch 2023-11-17T05:35:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.566260+0000 mon.smithi006 (mon.0) 307 : audit [INF] from='client.? 172.21.15.6:0/2410430036' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e3f232f-34c0-4619-b611-c288652cf48a"}]': finished 2023-11-17T05:35:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: cluster 2023-11-17T05:35:26.566329+0000 mon.smithi006 (mon.0) 308 : cluster [DBG] osdmap e11: 6 total, 0 up, 6 in 2023-11-17T05:35:27.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.566532+0000 mon.smithi006 (mon.0) 309 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.567354+0000 mon.smithi006 (mon.0) 310 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:27.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.567943+0000 mon.smithi006 (mon.0) 311 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:27.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.568454+0000 mon.smithi006 (mon.0) 312 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:27.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.568995+0000 mon.smithi006 (mon.0) 313 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:27.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:27 smithi006 bash[26022]: audit 2023-11-17T05:35:26.569509+0000 mon.smithi006 (mon.0) 314 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: cluster 2023-11-17T05:35:26.501118+0000 mgr.smithi006.odexej (mgr.14176) 100 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.561150+0000 mon.smithi006 (mon.0) 306 : audit [INF] from='client.? 172.21.15.6:0/2410430036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e3f232f-34c0-4619-b611-c288652cf48a"}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.566260+0000 mon.smithi006 (mon.0) 307 : audit [INF] from='client.? 172.21.15.6:0/2410430036' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e3f232f-34c0-4619-b611-c288652cf48a"}]': finished 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: cluster 2023-11-17T05:35:26.566329+0000 mon.smithi006 (mon.0) 308 : cluster [DBG] osdmap e11: 6 total, 0 up, 6 in 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.566532+0000 mon.smithi006 (mon.0) 309 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.567354+0000 mon.smithi006 (mon.0) 310 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.567943+0000 mon.smithi006 (mon.0) 311 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.568454+0000 mon.smithi006 (mon.0) 312 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.568995+0000 mon.smithi006 (mon.0) 313 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:27 smithi156 bash[29655]: audit 2023-11-17T05:35:26.569509+0000 mon.smithi006 (mon.0) 314 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:28.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.622099+0000 mon.smithi156 (mon.1) 8 : audit [INF] from='client.? 172.21.15.156:0/2826139555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]: dispatch 2023-11-17T05:35:28.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.623234+0000 mon.smithi006 (mon.0) 315 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]: dispatch 2023-11-17T05:35:28.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.630121+0000 mon.smithi006 (mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]': finished 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: cluster 2023-11-17T05:35:27.630263+0000 mon.smithi006 (mon.0) 317 : cluster [DBG] osdmap e12: 7 total, 0 up, 7 in 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.630498+0000 mon.smithi006 (mon.0) 318 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.631149+0000 mon.smithi006 (mon.0) 319 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.631503+0000 mon.smithi006 (mon.0) 320 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.631936+0000 mon.smithi006 (mon.0) 321 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.632361+0000 mon.smithi006 (mon.0) 322 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.632730+0000 mon.smithi006 (mon.0) 323 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:27.632943+0000 mon.smithi006 (mon.0) 324 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:35:28.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:28 smithi006 bash[26022]: audit 2023-11-17T05:35:28.032273+0000 mon.smithi006 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.6:0/1202846995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.622099+0000 mon.smithi156 (mon.1) 8 : audit [INF] from='client.? 172.21.15.156:0/2826139555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.623234+0000 mon.smithi006 (mon.0) 315 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.630121+0000 mon.smithi006 (mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab650abe-2841-4f3b-93e6-76be0832bfb6"}]': finished 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: cluster 2023-11-17T05:35:27.630263+0000 mon.smithi006 (mon.0) 317 : cluster [DBG] osdmap e12: 7 total, 0 up, 7 in 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.630498+0000 mon.smithi006 (mon.0) 318 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.631149+0000 mon.smithi006 (mon.0) 319 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.631503+0000 mon.smithi006 (mon.0) 320 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.631936+0000 mon.smithi006 (mon.0) 321 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.632361+0000 mon.smithi006 (mon.0) 322 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.632730+0000 mon.smithi006 (mon.0) 323 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:27.632943+0000 mon.smithi006 (mon.0) 324 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:35:28.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:28 smithi156 bash[29655]: audit 2023-11-17T05:35:28.032273+0000 mon.smithi006 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.6:0/1202846995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:28.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:29.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:29 smithi006 bash[26022]: cluster 2023-11-17T05:35:28.501431+0000 mgr.smithi006.odexej (mgr.14176) 101 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:29.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:29 smithi006 bash[26022]: audit 2023-11-17T05:35:28.640791+0000 mon.smithi156 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.156:0/15201863' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:29.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:29 smithi006 bash[26022]: audit 2023-11-17T05:35:28.866325+0000 mon.smithi006 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.6:0/2900805850' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:29.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:29 smithi156 bash[29655]: cluster 2023-11-17T05:35:28.501431+0000 mgr.smithi006.odexej (mgr.14176) 101 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:29.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:29 smithi156 bash[29655]: audit 2023-11-17T05:35:28.640791+0000 mon.smithi156 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.156:0/15201863' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:29.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:29 smithi156 bash[29655]: audit 2023-11-17T05:35:28.866325+0000 mon.smithi006 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.6:0/2900805850' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:29.493 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":12,"num_osds":7,"num_up_osds":0,"osd_up_since":0,"num_in_osds":7,"osd_in_since":1700199327,"num_remapped_pgs":0} 2023-11-17T05:35:30.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:31.864 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: cluster 2023-11-17T05:35:30.501872+0000 mgr.smithi006.odexej (mgr.14176) 102 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.806174+0000 mon.smithi006 (mon.0) 327 : audit [INF] from='client.? 172.21.15.6:0/458313340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80c4e482-5a64-46d6-9ad0-48b038823b84"}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.815095+0000 mon.smithi006 (mon.0) 328 : audit [INF] from='client.? 172.21.15.6:0/458313340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80c4e482-5a64-46d6-9ad0-48b038823b84"}]': finished 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: cluster 2023-11-17T05:35:30.815271+0000 mon.smithi006 (mon.0) 329 : cluster [DBG] osdmap e13: 8 total, 0 up, 8 in 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.815461+0000 mon.smithi006 (mon.0) 330 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.815932+0000 mon.smithi006 (mon.0) 331 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.816163+0000 mon.smithi006 (mon.0) 332 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.816424+0000 mon.smithi006 (mon.0) 333 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.816737+0000 mon.smithi006 (mon.0) 334 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.817064+0000 mon.smithi006 (mon.0) 335 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.817327+0000 mon.smithi006 (mon.0) 336 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:35:31.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:31 smithi006 bash[26022]: audit 2023-11-17T05:35:30.817680+0000 mon.smithi006 (mon.0) 337 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: cluster 2023-11-17T05:35:30.501872+0000 mgr.smithi006.odexej (mgr.14176) 102 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.806174+0000 mon.smithi006 (mon.0) 327 : audit [INF] from='client.? 172.21.15.6:0/458313340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80c4e482-5a64-46d6-9ad0-48b038823b84"}]: dispatch 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.815095+0000 mon.smithi006 (mon.0) 328 : audit [INF] from='client.? 172.21.15.6:0/458313340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80c4e482-5a64-46d6-9ad0-48b038823b84"}]': finished 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: cluster 2023-11-17T05:35:30.815271+0000 mon.smithi006 (mon.0) 329 : cluster [DBG] osdmap e13: 8 total, 0 up, 8 in 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.815461+0000 mon.smithi006 (mon.0) 330 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:35:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.815932+0000 mon.smithi006 (mon.0) 331 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:35:31.998 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.816163+0000 mon.smithi006 (mon.0) 332 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:35:31.999 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.816424+0000 mon.smithi006 (mon.0) 333 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:35:31.999 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.816737+0000 mon.smithi006 (mon.0) 334 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:35:32.000 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.817064+0000 mon.smithi006 (mon.0) 335 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:35:32.001 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.817327+0000 mon.smithi006 (mon.0) 336 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:35:32.001 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:31 smithi156 bash[29655]: audit 2023-11-17T05:35:30.817680+0000 mon.smithi006 (mon.0) 337 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:35:32.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:32 smithi006 bash[26022]: audit 2023-11-17T05:35:32.343065+0000 mon.smithi006 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.6:0/4293536677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:32.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:32 smithi156 bash[29655]: audit 2023-11-17T05:35:32.343065+0000 mon.smithi006 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.6:0/4293536677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T05:35:33.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:33 smithi006 bash[26022]: cluster 2023-11-17T05:35:32.502352+0000 mgr.smithi006.odexej (mgr.14176) 103 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:33.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:35:33.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:33 smithi156 bash[29655]: cluster 2023-11-17T05:35:32.502352+0000 mgr.smithi006.odexej (mgr.14176) 103 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:34.417 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":13,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1700199330,"num_remapped_pgs":0} 2023-11-17T05:35:34.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:34 smithi006 bash[26022]: audit 2023-11-17T05:35:33.787504+0000 mon.smithi006 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.6:0/2137388478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:34.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:34 smithi156 bash[29655]: audit 2023-11-17T05:35:33.787504+0000 mon.smithi006 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.6:0/2137388478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:35:35.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:35:35.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:35 smithi006 bash[26022]: cluster 2023-11-17T05:35:34.502797+0000 mgr.smithi006.odexej (mgr.14176) 104 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:35.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:35 smithi156 bash[29655]: cluster 2023-11-17T05:35:34.502797+0000 mgr.smithi006.odexej (mgr.14176) 104 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:37.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:37 smithi006 bash[26022]: cluster 2023-11-17T05:35:36.503292+0000 mgr.smithi006.odexej (mgr.14176) 105 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:37.429 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:37 smithi156 bash[29655]: cluster 2023-11-17T05:35:36.503292+0000 mgr.smithi006.odexej (mgr.14176) 105 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:38.395 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:36:24.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:38 smithi006 bash[26022]: audit 2023-11-17T05:35:38.394509+0000 mon.smithi006 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.6:0/1476482419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:36:25.013 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":13,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1700199330,"num_remapped_pgs":0} 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:38 smithi156 bash[29655]: audit 2023-11-17T05:35:38.394509+0000 mon.smithi006 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.6:0/1476482419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:39 smithi156 bash[29655]: cluster 2023-11-17T05:35:38.503820+0000 mgr.smithi006.odexej (mgr.14176) 106 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:40 smithi156 bash[29655]: audit 2023-11-17T05:35:39.983881+0000 mon.smithi006 (mon.0) 341 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:40 smithi156 bash[29655]: audit 2023-11-17T05:35:39.986510+0000 mon.smithi006 (mon.0) 342 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:41 smithi156 bash[29655]: cephadm 2023-11-17T05:35:39.987944+0000 mgr.smithi006.odexej (mgr.14176) 107 : cephadm [INF] Deploying daemon osd.0 on smithi156 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:41 smithi156 bash[29655]: cluster 2023-11-17T05:35:40.504292+0000 mgr.smithi006.odexej (mgr.14176) 108 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:43 smithi156 bash[29655]: cluster 2023-11-17T05:35:42.504822+0000 mgr.smithi006.odexej (mgr.14176) 109 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: audit 2023-11-17T05:35:43.670654+0000 mon.smithi006 (mon.0) 343 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: audit 2023-11-17T05:35:43.672312+0000 mon.smithi006 (mon.0) 344 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: audit 2023-11-17T05:35:43.673975+0000 mon.smithi006 (mon.0) 345 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: cephadm 2023-11-17T05:35:43.675556+0000 mgr.smithi006.odexej (mgr.14176) 110 : cephadm [INF] Deploying daemon osd.2 on smithi156 2023-11-17T05:36:25.020 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: audit 2023-11-17T05:35:43.876285+0000 mon.smithi006 (mon.0) 346 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:44 smithi156 bash[29655]: audit 2023-11-17T05:35:43.877711+0000 mon.smithi006 (mon.0) 347 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:45 smithi156 bash[29655]: cephadm 2023-11-17T05:35:43.878991+0000 mgr.smithi006.odexej (mgr.14176) 111 : cephadm [INF] Deploying daemon osd.1 on smithi006 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:45 smithi156 bash[29655]: cluster 2023-11-17T05:35:44.505256+0000 mgr.smithi006.odexej (mgr.14176) 112 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:47 smithi156 bash[29655]: cluster 2023-11-17T05:35:46.505753+0000 mgr.smithi006.odexej (mgr.14176) 113 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:47.821821+0000 mon.smithi006 (mon.0) 348 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:47.823264+0000 mon.smithi006 (mon.0) 349 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:47.824591+0000 mon.smithi006 (mon.0) 350 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: cephadm 2023-11-17T05:35:47.825583+0000 mgr.smithi006.odexej (mgr.14176) 114 : cephadm [INF] Deploying daemon osd.3 on smithi006 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:48.093799+0000 mon.smithi006 (mon.0) 351 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:48.095161+0000 mon.smithi006 (mon.0) 352 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:48 smithi156 bash[29655]: audit 2023-11-17T05:35:48.096562+0000 mon.smithi006 (mon.0) 353 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:49 smithi156 bash[29655]: cephadm 2023-11-17T05:35:48.097763+0000 mgr.smithi006.odexej (mgr.14176) 115 : cephadm [INF] Deploying daemon osd.4 on smithi156 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:49 smithi156 bash[29655]: cluster 2023-11-17T05:35:48.506187+0000 mgr.smithi006.odexej (mgr.14176) 116 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.021 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:51 smithi156 bash[29655]: cluster 2023-11-17T05:35:50.506655+0000 mgr.smithi006.odexej (mgr.14176) 117 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:51 smithi156 bash[29655]: audit 2023-11-17T05:35:51.492288+0000 mon.smithi156 (mon.1) 10 : audit [INF] from='osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:51 smithi156 bash[29655]: audit 2023-11-17T05:35:51.493328+0000 mon.smithi006 (mon.0) 354 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.149275+0000 mon.smithi006 (mon.0) 355 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: cluster 2023-11-17T05:35:52.149437+0000 mon.smithi006 (mon.0) 356 : cluster [DBG] osdmap e14: 8 total, 0 up, 8 in 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.149668+0000 mon.smithi006 (mon.0) 357 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.150360+0000 mon.smithi156 (mon.1) 11 : audit [INF] from='osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.150408+0000 mon.smithi006 (mon.0) 358 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.150828+0000 mon.smithi006 (mon.0) 359 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.151449+0000 mon.smithi006 (mon.0) 360 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.152194+0000 mon.smithi006 (mon.0) 361 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.152745+0000 mon.smithi006 (mon.0) 362 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.153163+0000 mon.smithi006 (mon.0) 363 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.022 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.153553+0000 mon.smithi006 (mon.0) 364 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.153966+0000 mon.smithi006 (mon.0) 365 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: cluster 2023-11-17T05:35:52.506987+0000 mgr.smithi006.odexej (mgr.14176) 118 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.611230+0000 mon.smithi006 (mon.0) 366 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.612687+0000 mon.smithi006 (mon.0) 367 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:52.613814+0000 mon.smithi006 (mon.0) 368 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: cephadm 2023-11-17T05:35:52.614708+0000 mgr.smithi006.odexej (mgr.14176) 119 : cephadm [INF] Deploying daemon osd.5 on smithi006 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:53.103158+0000 mon.smithi006 (mon.0) 369 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:53.105102+0000 mon.smithi006 (mon.0) 370 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:53 smithi156 bash[29655]: audit 2023-11-17T05:35:53.106431+0000 mon.smithi006 (mon.0) 371 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: cephadm 2023-11-17T05:35:53.107602+0000 mgr.smithi006.odexej (mgr.14176) 120 : cephadm [INF] Deploying daemon osd.6 on smithi156 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.160097+0000 mon.smithi006 (mon.0) 372 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: cluster 2023-11-17T05:35:53.160233+0000 mon.smithi006 (mon.0) 373 : cluster [DBG] osdmap e15: 8 total, 0 up, 8 in 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.160954+0000 mon.smithi006 (mon.0) 374 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.023 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.161776+0000 mon.smithi006 (mon.0) 375 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.162089+0000 mon.smithi006 (mon.0) 376 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.162432+0000 mon.smithi006 (mon.0) 377 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.162717+0000 mon.smithi006 (mon.0) 378 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.163051+0000 mon.smithi006 (mon.0) 379 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.163336+0000 mon.smithi006 (mon.0) 380 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.163706+0000 mon.smithi006 (mon.0) 381 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:54 smithi156 bash[29655]: audit 2023-11-17T05:35:53.166222+0000 mon.smithi006 (mon.0) 382 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: cluster 2023-11-17T05:35:52.485408+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: cluster 2023-11-17T05:35:52.485496+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: cluster 2023-11-17T05:35:54.159374+0000 mon.smithi006 (mon.0) 383 : cluster [INF] osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703] boot 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: cluster 2023-11-17T05:35:54.159530+0000 mon.smithi006 (mon.0) 384 : cluster [DBG] osdmap e16: 8 total, 1 up, 8 in 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.160356+0000 mon.smithi006 (mon.0) 385 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.161502+0000 mon.smithi006 (mon.0) 386 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.024 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.168912+0000 mon.smithi006 (mon.0) 387 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.170213+0000 mon.smithi006 (mon.0) 388 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.170895+0000 mon.smithi006 (mon.0) 389 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.171614+0000 mon.smithi006 (mon.0) 390 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.172201+0000 mon.smithi006 (mon.0) 391 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: audit 2023-11-17T05:35:54.172608+0000 mon.smithi006 (mon.0) 392 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:55 smithi156 bash[29655]: cluster 2023-11-17T05:35:54.507266+0000 mgr.smithi006.odexej (mgr.14176) 121 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: cluster 2023-11-17T05:35:55.177963+0000 mon.smithi006 (mon.0) 393 : cluster [DBG] osdmap e17: 8 total, 1 up, 8 in 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.178212+0000 mon.smithi006 (mon.0) 394 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.178735+0000 mon.smithi006 (mon.0) 395 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.179109+0000 mon.smithi006 (mon.0) 396 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.179383+0000 mon.smithi006 (mon.0) 397 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.179694+0000 mon.smithi006 (mon.0) 398 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.180016+0000 mon.smithi006 (mon.0) 399 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.025 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.180323+0000 mon.smithi006 (mon.0) 400 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.898861+0000 mon.smithi006 (mon.0) 401 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.945578+0000 mon.smithi156 (mon.1) 12 : audit [INF] from='osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:56 smithi156 bash[29655]: audit 2023-11-17T05:35:55.946558+0000 mon.smithi006 (mon.0) 402 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.188045+0000 mon.smithi006 (mon.0) 403 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.188268+0000 mon.smithi006 (mon.0) 404 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.188420+0000 mon.smithi006 (mon.0) 405 : cluster [DBG] osdmap e18: 8 total, 1 up, 8 in 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.188613+0000 mon.smithi006 (mon.0) 406 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.188995+0000 mon.smithi156 (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.189122+0000 mon.smithi006 (mon.0) 407 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.189419+0000 mon.smithi006 (mon.0) 408 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.189873+0000 mon.smithi006 (mon.0) 409 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.190172+0000 mon.smithi006 (mon.0) 410 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.190454+0000 mon.smithi006 (mon.0) 411 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.026 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.190661+0000 mon.smithi006 (mon.0) 412 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.190885+0000 mon.smithi006 (mon.0) 413 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: audit 2023-11-17T05:35:56.191122+0000 mon.smithi006 (mon.0) 414 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:57 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.507565+0000 mgr.smithi006.odexej (mgr.14176) 122 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.189365+0000 mon.smithi006 (mon.0) 415 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.189485+0000 mon.smithi006 (mon.0) 416 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: cluster 2023-11-17T05:35:57.189585+0000 mon.smithi006 (mon.0) 417 : cluster [DBG] osdmap e19: 8 total, 1 up, 8 in 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.191327+0000 mon.smithi006 (mon.0) 418 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.192215+0000 mon.smithi006 (mon.0) 419 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.193262+0000 mon.smithi006 (mon.0) 420 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.193983+0000 mon.smithi006 (mon.0) 421 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.194720+0000 mon.smithi006 (mon.0) 422 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.195147+0000 mon.smithi006 (mon.0) 423 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.027 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.195622+0000 mon.smithi006 (mon.0) 424 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.200953+0000 mon.smithi006 (mon.0) 425 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:57.201585+0000 mon.smithi006 (mon.0) 426 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:58 smithi156 bash[29655]: audit 2023-11-17T05:35:58.195533+0000 mon.smithi006 (mon.0) 427 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.905453+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.905537+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.946982+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:56.947044+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.197606+0000 mon.smithi006 (mon.0) 428 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:58.203766+0000 mon.smithi006 (mon.0) 429 : cluster [INF] osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933] boot 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:58.203806+0000 mon.smithi006 (mon.0) 430 : cluster [INF] osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296] boot 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:58.203841+0000 mon.smithi006 (mon.0) 431 : cluster [DBG] osdmap e20: 8 total, 3 up, 8 in 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.204390+0000 mon.smithi006 (mon.0) 432 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.204657+0000 mon.smithi006 (mon.0) 433 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.028 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.205011+0000 mon.smithi006 (mon.0) 434 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.205145+0000 mon.smithi006 (mon.0) 435 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.210281+0000 mon.smithi006 (mon.0) 436 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.210526+0000 mon.smithi006 (mon.0) 437 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.210832+0000 mon.smithi006 (mon.0) 438 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.211151+0000 mon.smithi006 (mon.0) 439 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cluster 2023-11-17T05:35:58.507794+0000 mgr.smithi006.odexej (mgr.14176) 123 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.684595+0000 mon.smithi006 (mon.0) 440 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.686407+0000 mon.smithi006 (mon.0) 441 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.690563+0000 mon.smithi006 (mon.0) 442 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: audit 2023-11-17T05:35:58.691073+0000 mon.smithi006 (mon.0) 443 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:35:59 smithi156 bash[29655]: cephadm 2023-11-17T05:35:58.692639+0000 mgr.smithi006.odexej (mgr.14176) 124 : cephadm [INF] Deploying daemon osd.7 on smithi006 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.218786+0000 mon.smithi006 (mon.0) 444 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: cluster 2023-11-17T05:35:59.218882+0000 mon.smithi006 (mon.0) 445 : cluster [DBG] osdmap e21: 8 total, 3 up, 8 in 2023-11-17T05:36:25.029 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.219081+0000 mon.smithi006 (mon.0) 446 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.219765+0000 mon.smithi006 (mon.0) 447 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.220204+0000 mon.smithi006 (mon.0) 448 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.220678+0000 mon.smithi006 (mon.0) 449 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.220998+0000 mon.smithi006 (mon.0) 450 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:00 smithi156 bash[29655]: audit 2023-11-17T05:35:59.221864+0000 mon.smithi006 (mon.0) 451 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-11-17T05:36:25.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.222010+0000 mon.smithi006 (mon.0) 452 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-11-17T05:36:25.052 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: cluster 2023-11-17T05:36:00.222184+0000 mon.smithi006 (mon.0) 453 : cluster [DBG] osdmap e22: 8 total, 3 up, 8 in 2023-11-17T05:36:25.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.222907+0000 mon.smithi006 (mon.0) 454 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.223329+0000 mon.smithi006 (mon.0) 455 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.223626+0000 mon.smithi006 (mon.0) 456 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.223930+0000 mon.smithi006 (mon.0) 457 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:00.224153+0000 mon.smithi006 (mon.0) 458 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: cluster 2023-11-17T05:36:00.508143+0000 mgr.smithi006.odexej (mgr.14176) 125 : cluster [DBG] pgmap v84: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T05:36:25.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:01 smithi156 bash[29655]: audit 2023-11-17T05:36:01.173947+0000 mon.smithi006 (mon.0) 459 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.235247+0000 mon.smithi006 (mon.0) 460 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: cluster 2023-11-17T05:36:01.235332+0000 mon.smithi006 (mon.0) 461 : cluster [DBG] osdmap e23: 8 total, 3 up, 8 in 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.235556+0000 mon.smithi006 (mon.0) 462 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.236366+0000 mon.smithi006 (mon.0) 463 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.237001+0000 mon.smithi006 (mon.0) 464 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.237596+0000 mon.smithi006 (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.238295+0000 mon.smithi006 (mon.0) 466 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.238841+0000 mon.smithi006 (mon.0) 467 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.622430+0000 mon.smithi156 (mon.1) 14 : audit [INF] from='osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:02 smithi156 bash[29655]: audit 2023-11-17T05:36:01.623389+0000 mon.smithi006 (mon.0) 468 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.239903+0000 mon.smithi006 (mon.0) 469 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.083 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.240112+0000 mon.smithi006 (mon.0) 470 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-17T05:36:25.083 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.240274+0000 mon.smithi006 (mon.0) 471 : cluster [DBG] osdmap e24: 8 total, 3 up, 8 in 2023-11-17T05:36:25.208 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.241289+0000 mon.smithi006 (mon.0) 472 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.241410+0000 mon.smithi156 (mon.1) 15 : audit [INF] from='osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.241956+0000 mon.smithi006 (mon.0) 473 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.243258+0000 mon.smithi006 (mon.0) 474 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.247998+0000 mon.smithi006 (mon.0) 475 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.248439+0000 mon.smithi006 (mon.0) 476 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.248768+0000 mon.smithi006 (mon.0) 477 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:02.249572+0000 mon.smithi006 (mon.0) 478 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.508643+0000 mgr.smithi006.odexej (mgr.14176) 126 : cluster [DBG] pgmap v87: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T05:36:25.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:03.242413+0000 mon.smithi006 (mon.0) 479 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: cluster 2023-11-17T05:36:03.242538+0000 mon.smithi006 (mon.0) 480 : cluster [INF] osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715] boot 2023-11-17T05:36:25.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: cluster 2023-11-17T05:36:03.242618+0000 mon.smithi006 (mon.0) 481 : cluster [DBG] osdmap e25: 8 total, 4 up, 8 in 2023-11-17T05:36:25.240 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:03 smithi156 bash[29655]: audit 2023-11-17T05:36:03.242818+0000 mon.smithi006 (mon.0) 482 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.240 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.214136+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.240 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.214259+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.240 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.244906+0000 mon.smithi006 (mon.0) 483 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.241 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.245239+0000 mon.smithi006 (mon.0) 484 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.241 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.245543+0000 mon.smithi006 (mon.0) 485 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.241 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.245838+0000 mon.smithi006 (mon.0) 486 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.241 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.251640+0000 mon.smithi006 (mon.0) 487 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.241 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:04 smithi156 bash[29655]: audit 2023-11-17T05:36:03.986050+0000 mon.smithi006 (mon.0) 488 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.655792+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:02.655887+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:04.245970+0000 mon.smithi006 (mon.0) 489 : cluster [INF] osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389] boot 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:04.246061+0000 mon.smithi006 (mon.0) 490 : cluster [DBG] osdmap e26: 8 total, 5 up, 8 in 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:04.246356+0000 mon.smithi006 (mon.0) 491 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:04.247683+0000 mon.smithi006 (mon.0) 492 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:04.247922+0000 mon.smithi006 (mon.0) 493 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:04.248203+0000 mon.smithi006 (mon.0) 494 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:04.508950+0000 mgr.smithi006.odexej (mgr.14176) 127 : cluster [DBG] pgmap v90: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.185265+0000 mon.smithi156 (mon.1) 16 : audit [INF] from='osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.186213+0000 mon.smithi006 (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T05:36:25.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.246841+0000 mon.smithi006 (mon.0) 496 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: cluster 2023-11-17T05:36:05.246960+0000 mon.smithi006 (mon.0) 497 : cluster [DBG] osdmap e27: 8 total, 5 up, 8 in 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.247138+0000 mon.smithi006 (mon.0) 498 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.247487+0000 mon.smithi006 (mon.0) 499 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:05 smithi156 bash[29655]: audit 2023-11-17T05:36:05.247706+0000 mon.smithi006 (mon.0) 500 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:05.248912+0000 mon.smithi156 (mon.1) 17 : audit [INF] from='osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.244 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:05.249741+0000 mon.smithi006 (mon.0) 501 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.245 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.241174+0000 mon.smithi006 (mon.0) 502 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-17T05:36:25.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.249282+0000 mon.smithi006 (mon.0) 503 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.249384+0000 mon.smithi006 (mon.0) 504 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-17T05:36:25.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: cluster 2023-11-17T05:36:06.249447+0000 mon.smithi006 (mon.0) 505 : cluster [DBG] osdmap e28: 8 total, 5 up, 8 in 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.249614+0000 mon.smithi006 (mon.0) 506 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.250062+0000 mon.smithi006 (mon.0) 507 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.250329+0000 mon.smithi006 (mon.0) 508 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:06 smithi156 bash[29655]: audit 2023-11-17T05:36:06.250605+0000 mon.smithi006 (mon.0) 509 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:07 smithi156 bash[29655]: audit 2023-11-17T05:36:06.256896+0000 mon.smithi006 (mon.0) 510 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:07 smithi156 bash[29655]: cluster 2023-11-17T05:36:06.509474+0000 mgr.smithi006.odexej (mgr.14176) 128 : cluster [DBG] pgmap v93: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: cluster 2023-11-17T05:36:06.205406+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: cluster 2023-11-17T05:36:06.205508+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.251910+0000 mon.smithi006 (mon.0) 511 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: cluster 2023-11-17T05:36:07.252072+0000 mon.smithi006 (mon.0) 512 : cluster [INF] osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943] boot 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: cluster 2023-11-17T05:36:07.252160+0000 mon.smithi006 (mon.0) 513 : cluster [DBG] osdmap e29: 8 total, 6 up, 8 in 2023-11-17T05:36:25.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.252476+0000 mon.smithi006 (mon.0) 514 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.254188+0000 mon.smithi006 (mon.0) 515 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.257784+0000 mon.smithi006 (mon.0) 516 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.263455+0000 mon.smithi006 (mon.0) 517 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:08 smithi156 bash[29655]: audit 2023-11-17T05:36:07.549005+0000 mon.smithi006 (mon.0) 518 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: cluster 2023-11-17T05:36:07.269468+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: cluster 2023-11-17T05:36:07.269959+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: audit 2023-11-17T05:36:08.269663+0000 mon.smithi006 (mon.0) 519 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: cluster 2023-11-17T05:36:08.275843+0000 mon.smithi006 (mon.0) 520 : cluster [INF] osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604] boot 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: cluster 2023-11-17T05:36:08.275910+0000 mon.smithi006 (mon.0) 521 : cluster [DBG] osdmap e30: 8 total, 7 up, 8 in 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: audit 2023-11-17T05:36:08.276052+0000 mon.smithi006 (mon.0) 522 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: audit 2023-11-17T05:36:08.276521+0000 mon.smithi006 (mon.0) 523 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:09 smithi156 bash[29655]: cluster 2023-11-17T05:36:08.510013+0000 mgr.smithi006.odexej (mgr.14176) 129 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T05:36:25.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:10 smithi156 bash[29655]: cluster 2023-11-17T05:36:09.281399+0000 mon.smithi006 (mon.0) 524 : cluster [DBG] osdmap e31: 8 total, 7 up, 8 in 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:10 smithi156 bash[29655]: audit 2023-11-17T05:36:09.281660+0000 mon.smithi006 (mon.0) 525 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:11 smithi156 bash[29655]: cluster 2023-11-17T05:36:10.510605+0000 mgr.smithi006.odexej (mgr.14176) 130 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:11 smithi156 bash[29655]: audit 2023-11-17T05:36:11.155988+0000 mon.smithi006 (mon.0) 526 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:13 smithi156 bash[29655]: audit 2023-11-17T05:36:12.153652+0000 mon.smithi006 (mon.0) 527 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:13 smithi156 bash[29655]: cluster 2023-11-17T05:36:12.153748+0000 mon.smithi006 (mon.0) 528 : cluster [DBG] osdmap e32: 8 total, 7 up, 8 in 2023-11-17T05:36:25.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:13 smithi156 bash[29655]: audit 2023-11-17T05:36:12.153910+0000 mon.smithi006 (mon.0) 529 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:13 smithi156 bash[29655]: audit 2023-11-17T05:36:12.154478+0000 mon.smithi006 (mon.0) 530 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:13 smithi156 bash[29655]: cluster 2023-11-17T05:36:12.511114+0000 mgr.smithi006.odexej (mgr.14176) 131 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: cluster 2023-11-17T05:36:12.140134+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: cluster 2023-11-17T05:36:12.140219+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: audit 2023-11-17T05:36:13.156824+0000 mon.smithi006 (mon.0) 531 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: cluster 2023-11-17T05:36:13.156952+0000 mon.smithi006 (mon.0) 532 : cluster [DBG] osdmap e33: 8 total, 7 up, 8 in 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: audit 2023-11-17T05:36:13.157558+0000 mon.smithi006 (mon.0) 533 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:14 smithi156 bash[29655]: audit 2023-11-17T05:36:13.164765+0000 mon.smithi006 (mon.0) 534 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: audit 2023-11-17T05:36:14.169144+0000 mon.smithi006 (mon.0) 535 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: cluster 2023-11-17T05:36:14.179288+0000 mon.smithi006 (mon.0) 536 : cluster [INF] osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079] boot 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: cluster 2023-11-17T05:36:14.179456+0000 mon.smithi006 (mon.0) 537 : cluster [DBG] osdmap e34: 8 total, 8 up, 8 in 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: audit 2023-11-17T05:36:14.180128+0000 mon.smithi006 (mon.0) 538 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: audit 2023-11-17T05:36:14.207007+0000 mon.smithi006 (mon.0) 539 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: audit 2023-11-17T05:36:14.210058+0000 mon.smithi006 (mon.0) 540 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:15 smithi156 bash[29655]: cluster 2023-11-17T05:36:14.511752+0000 mgr.smithi006.odexej (mgr.14176) 132 : cluster [DBG] pgmap v103: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:16 smithi156 bash[29655]: cluster 2023-11-17T05:36:15.177779+0000 mon.smithi006 (mon.0) 541 : cluster [DBG] osdmap e35: 8 total, 8 up, 8 in 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:17 smithi156 bash[29655]: cluster 2023-11-17T05:36:16.190716+0000 mon.smithi006 (mon.0) 542 : cluster [DBG] osdmap e36: 8 total, 8 up, 8 in 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:17 smithi156 bash[29655]: cluster 2023-11-17T05:36:16.512244+0000 mgr.smithi006.odexej (mgr.14176) 133 : cluster [DBG] pgmap v106: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:19 smithi156 bash[29655]: cluster 2023-11-17T05:36:18.512825+0000 mgr.smithi006.odexej (mgr.14176) 134 : cluster [DBG] pgmap v107: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:25.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:20 smithi156 bash[29655]: audit 2023-11-17T05:36:19.852134+0000 mon.smithi006 (mon.0) 543 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.380 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:20 smithi156 bash[29655]: audit 2023-11-17T05:36:20.499892+0000 mon.smithi006 (mon.0) 544 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.380 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:21 smithi156 bash[29655]: cluster 2023-11-17T05:36:20.513376+0000 mgr.smithi006.odexej (mgr.14176) 135 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:25.380 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:23 smithi156 bash[29655]: cluster 2023-11-17T05:36:22.513933+0000 mgr.smithi006.odexej (mgr.14176) 136 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:25.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:39 smithi006 bash[26022]: cluster 2023-11-17T05:35:38.503820+0000 mgr.smithi006.odexej (mgr.14176) 106 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:40 smithi006 bash[26022]: audit 2023-11-17T05:35:39.983881+0000 mon.smithi006 (mon.0) 341 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-17T05:36:25.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:40 smithi006 bash[26022]: audit 2023-11-17T05:35:39.986510+0000 mon.smithi006 (mon.0) 342 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:41 smithi006 bash[26022]: cephadm 2023-11-17T05:35:39.987944+0000 mgr.smithi006.odexej (mgr.14176) 107 : cephadm [INF] Deploying daemon osd.0 on smithi156 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:41 smithi006 bash[26022]: cluster 2023-11-17T05:35:40.504292+0000 mgr.smithi006.odexej (mgr.14176) 108 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:43 smithi006 bash[26022]: cluster 2023-11-17T05:35:42.504822+0000 mgr.smithi006.odexej (mgr.14176) 109 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: audit 2023-11-17T05:35:43.670654+0000 mon.smithi006 (mon.0) 343 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: audit 2023-11-17T05:35:43.672312+0000 mon.smithi006 (mon.0) 344 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: audit 2023-11-17T05:35:43.673975+0000 mon.smithi006 (mon.0) 345 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: cephadm 2023-11-17T05:35:43.675556+0000 mgr.smithi006.odexej (mgr.14176) 110 : cephadm [INF] Deploying daemon osd.2 on smithi156 2023-11-17T05:36:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: audit 2023-11-17T05:35:43.876285+0000 mon.smithi006 (mon.0) 346 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:44 smithi006 bash[26022]: audit 2023-11-17T05:35:43.877711+0000 mon.smithi006 (mon.0) 347 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:45 smithi006 bash[26022]: cephadm 2023-11-17T05:35:43.878991+0000 mgr.smithi006.odexej (mgr.14176) 111 : cephadm [INF] Deploying daemon osd.1 on smithi006 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:45 smithi006 bash[26022]: cluster 2023-11-17T05:35:44.505256+0000 mgr.smithi006.odexej (mgr.14176) 112 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:47 smithi006 bash[26022]: cluster 2023-11-17T05:35:46.505753+0000 mgr.smithi006.odexej (mgr.14176) 113 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:47.821821+0000 mon.smithi006 (mon.0) 348 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:47.823264+0000 mon.smithi006 (mon.0) 349 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:47.824591+0000 mon.smithi006 (mon.0) 350 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: cephadm 2023-11-17T05:35:47.825583+0000 mgr.smithi006.odexej (mgr.14176) 114 : cephadm [INF] Deploying daemon osd.3 on smithi006 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:48.093799+0000 mon.smithi006 (mon.0) 351 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:48.095161+0000 mon.smithi006 (mon.0) 352 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:48 smithi006 bash[26022]: audit 2023-11-17T05:35:48.096562+0000 mon.smithi006 (mon.0) 353 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:49 smithi006 bash[26022]: cephadm 2023-11-17T05:35:48.097763+0000 mgr.smithi006.odexej (mgr.14176) 115 : cephadm [INF] Deploying daemon osd.4 on smithi156 2023-11-17T05:36:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:49 smithi006 bash[26022]: cluster 2023-11-17T05:35:48.506187+0000 mgr.smithi006.odexej (mgr.14176) 116 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:51 smithi006 bash[26022]: cluster 2023-11-17T05:35:50.506655+0000 mgr.smithi006.odexej (mgr.14176) 117 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:51 smithi006 bash[26022]: audit 2023-11-17T05:35:51.492288+0000 mon.smithi156 (mon.1) 10 : audit [INF] from='osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:51 smithi006 bash[26022]: audit 2023-11-17T05:35:51.493328+0000 mon.smithi006 (mon.0) 354 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.149275+0000 mon.smithi006 (mon.0) 355 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: cluster 2023-11-17T05:35:52.149437+0000 mon.smithi006 (mon.0) 356 : cluster [DBG] osdmap e14: 8 total, 0 up, 8 in 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.149668+0000 mon.smithi006 (mon.0) 357 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.150360+0000 mon.smithi156 (mon.1) 11 : audit [INF] from='osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.150408+0000 mon.smithi006 (mon.0) 358 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.150828+0000 mon.smithi006 (mon.0) 359 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.151449+0000 mon.smithi006 (mon.0) 360 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.152194+0000 mon.smithi006 (mon.0) 361 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.152745+0000 mon.smithi006 (mon.0) 362 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.153163+0000 mon.smithi006 (mon.0) 363 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.153553+0000 mon.smithi006 (mon.0) 364 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.153966+0000 mon.smithi006 (mon.0) 365 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: cluster 2023-11-17T05:35:52.506987+0000 mgr.smithi006.odexej (mgr.14176) 118 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.611230+0000 mon.smithi006 (mon.0) 366 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.612687+0000 mon.smithi006 (mon.0) 367 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:52.613814+0000 mon.smithi006 (mon.0) 368 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: cephadm 2023-11-17T05:35:52.614708+0000 mgr.smithi006.odexej (mgr.14176) 119 : cephadm [INF] Deploying daemon osd.5 on smithi006 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:53.103158+0000 mon.smithi006 (mon.0) 369 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:53.105102+0000 mon.smithi006 (mon.0) 370 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:53 smithi006 bash[26022]: audit 2023-11-17T05:35:53.106431+0000 mon.smithi006 (mon.0) 371 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: cephadm 2023-11-17T05:35:53.107602+0000 mgr.smithi006.odexej (mgr.14176) 120 : cephadm [INF] Deploying daemon osd.6 on smithi156 2023-11-17T05:36:25.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.160097+0000 mon.smithi006 (mon.0) 372 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.536 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: cluster 2023-11-17T05:35:53.160233+0000 mon.smithi006 (mon.0) 373 : cluster [DBG] osdmap e15: 8 total, 0 up, 8 in 2023-11-17T05:36:25.536 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.160954+0000 mon.smithi006 (mon.0) 374 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.536 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.161776+0000 mon.smithi006 (mon.0) 375 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.536 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.162089+0000 mon.smithi006 (mon.0) 376 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.162432+0000 mon.smithi006 (mon.0) 377 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.162717+0000 mon.smithi006 (mon.0) 378 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.163051+0000 mon.smithi006 (mon.0) 379 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.163336+0000 mon.smithi006 (mon.0) 380 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.163706+0000 mon.smithi006 (mon.0) 381 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:54 smithi006 bash[26022]: audit 2023-11-17T05:35:53.166222+0000 mon.smithi006 (mon.0) 382 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: cluster 2023-11-17T05:35:52.485408+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: cluster 2023-11-17T05:35:52.485496+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: cluster 2023-11-17T05:35:54.159374+0000 mon.smithi006 (mon.0) 383 : cluster [INF] osd.0 [v2:172.21.15.156:6800/3889302703,v1:172.21.15.156:6801/3889302703] boot 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: cluster 2023-11-17T05:35:54.159530+0000 mon.smithi006 (mon.0) 384 : cluster [DBG] osdmap e16: 8 total, 1 up, 8 in 2023-11-17T05:36:25.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.160356+0000 mon.smithi006 (mon.0) 385 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T05:36:25.539 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.161502+0000 mon.smithi006 (mon.0) 386 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.539 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.168912+0000 mon.smithi006 (mon.0) 387 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.539 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.170213+0000 mon.smithi006 (mon.0) 388 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.539 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.170895+0000 mon.smithi006 (mon.0) 389 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.539 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.171614+0000 mon.smithi006 (mon.0) 390 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.172201+0000 mon.smithi006 (mon.0) 391 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: audit 2023-11-17T05:35:54.172608+0000 mon.smithi006 (mon.0) 392 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:55 smithi006 bash[26022]: cluster 2023-11-17T05:35:54.507266+0000 mgr.smithi006.odexej (mgr.14176) 121 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: cluster 2023-11-17T05:35:55.177963+0000 mon.smithi006 (mon.0) 393 : cluster [DBG] osdmap e17: 8 total, 1 up, 8 in 2023-11-17T05:36:25.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.178212+0000 mon.smithi006 (mon.0) 394 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.541 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.178735+0000 mon.smithi006 (mon.0) 395 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.541 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.179109+0000 mon.smithi006 (mon.0) 396 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.541 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.179383+0000 mon.smithi006 (mon.0) 397 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.541 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.179694+0000 mon.smithi006 (mon.0) 398 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.541 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.180016+0000 mon.smithi006 (mon.0) 399 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.542 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.180323+0000 mon.smithi006 (mon.0) 400 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.898861+0000 mon.smithi006 (mon.0) 401 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-17T05:36:25.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.945578+0000 mon.smithi156 (mon.1) 12 : audit [INF] from='osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T05:36:25.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:56 smithi006 bash[26022]: audit 2023-11-17T05:35:55.946558+0000 mon.smithi006 (mon.0) 402 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T05:36:25.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.188045+0000 mon.smithi006 (mon.0) 403 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-17T05:36:25.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.188268+0000 mon.smithi006 (mon.0) 404 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-17T05:36:25.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.188420+0000 mon.smithi006 (mon.0) 405 : cluster [DBG] osdmap e18: 8 total, 1 up, 8 in 2023-11-17T05:36:25.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.188613+0000 mon.smithi006 (mon.0) 406 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.188995+0000 mon.smithi156 (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.189122+0000 mon.smithi006 (mon.0) 407 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.189419+0000 mon.smithi006 (mon.0) 408 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.189873+0000 mon.smithi006 (mon.0) 409 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.190172+0000 mon.smithi006 (mon.0) 410 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.190454+0000 mon.smithi006 (mon.0) 411 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.190661+0000 mon.smithi006 (mon.0) 412 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.190885+0000 mon.smithi006 (mon.0) 413 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: audit 2023-11-17T05:35:56.191122+0000 mon.smithi006 (mon.0) 414 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:57 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.507565+0000 mgr.smithi006.odexej (mgr.14176) 122 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.189365+0000 mon.smithi006 (mon.0) 415 : audit [INF] from='osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.593 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.189485+0000 mon.smithi006 (mon.0) 416 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.593 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: cluster 2023-11-17T05:35:57.189585+0000 mon.smithi006 (mon.0) 417 : cluster [DBG] osdmap e19: 8 total, 1 up, 8 in 2023-11-17T05:36:25.593 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.191327+0000 mon.smithi006 (mon.0) 418 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.593 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.192215+0000 mon.smithi006 (mon.0) 419 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.594 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.193262+0000 mon.smithi006 (mon.0) 420 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.594 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.193983+0000 mon.smithi006 (mon.0) 421 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.594 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.194720+0000 mon.smithi006 (mon.0) 422 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.594 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.195147+0000 mon.smithi006 (mon.0) 423 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.594 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.195622+0000 mon.smithi006 (mon.0) 424 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.596 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.200953+0000 mon.smithi006 (mon.0) 425 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.596 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:57.201585+0000 mon.smithi006 (mon.0) 426 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.596 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:58 smithi006 bash[26022]: audit 2023-11-17T05:35:58.195533+0000 mon.smithi006 (mon.0) 427 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.596 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.905453+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.905537+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.946982+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:56.947044+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.197606+0000 mon.smithi006 (mon.0) 428 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:58.203766+0000 mon.smithi006 (mon.0) 429 : cluster [INF] osd.2 [v2:172.21.15.156:6808/533925933,v1:172.21.15.156:6809/533925933] boot 2023-11-17T05:36:25.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:58.203806+0000 mon.smithi006 (mon.0) 430 : cluster [INF] osd.1 [v2:172.21.15.6:6802/3517964296,v1:172.21.15.6:6803/3517964296] boot 2023-11-17T05:36:25.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:58.203841+0000 mon.smithi006 (mon.0) 431 : cluster [DBG] osdmap e20: 8 total, 3 up, 8 in 2023-11-17T05:36:25.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.204390+0000 mon.smithi006 (mon.0) 432 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T05:36:25.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.204657+0000 mon.smithi006 (mon.0) 433 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.205011+0000 mon.smithi006 (mon.0) 434 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.205145+0000 mon.smithi006 (mon.0) 435 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.210281+0000 mon.smithi006 (mon.0) 436 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.210526+0000 mon.smithi006 (mon.0) 437 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.210832+0000 mon.smithi006 (mon.0) 438 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.623 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.211151+0000 mon.smithi006 (mon.0) 439 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-11-17T05:36:25.624 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cluster 2023-11-17T05:35:58.507794+0000 mgr.smithi006.odexej (mgr.14176) 123 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T05:36:25.624 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.684595+0000 mon.smithi006 (mon.0) 440 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.649 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.686407+0000 mon.smithi006 (mon.0) 441 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-17T05:36:25.649 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.690563+0000 mon.smithi006 (mon.0) 442 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: audit 2023-11-17T05:35:58.691073+0000 mon.smithi006 (mon.0) 443 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:35:59 smithi006 bash[26022]: cephadm 2023-11-17T05:35:58.692639+0000 mgr.smithi006.odexej (mgr.14176) 124 : cephadm [INF] Deploying daemon osd.7 on smithi006 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.218786+0000 mon.smithi006 (mon.0) 444 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: cluster 2023-11-17T05:35:59.218882+0000 mon.smithi006 (mon.0) 445 : cluster [DBG] osdmap e21: 8 total, 3 up, 8 in 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.219081+0000 mon.smithi006 (mon.0) 446 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.219765+0000 mon.smithi006 (mon.0) 447 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.220204+0000 mon.smithi006 (mon.0) 448 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.220678+0000 mon.smithi006 (mon.0) 449 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.220998+0000 mon.smithi006 (mon.0) 450 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.650 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:00 smithi006 bash[26022]: audit 2023-11-17T05:35:59.221864+0000 mon.smithi006 (mon.0) 451 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.222010+0000 mon.smithi006 (mon.0) 452 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: cluster 2023-11-17T05:36:00.222184+0000 mon.smithi006 (mon.0) 453 : cluster [DBG] osdmap e22: 8 total, 3 up, 8 in 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.222907+0000 mon.smithi006 (mon.0) 454 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.223329+0000 mon.smithi006 (mon.0) 455 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.223626+0000 mon.smithi006 (mon.0) 456 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.223930+0000 mon.smithi006 (mon.0) 457 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:00.224153+0000 mon.smithi006 (mon.0) 458 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: cluster 2023-11-17T05:36:00.508143+0000 mgr.smithi006.odexej (mgr.14176) 125 : cluster [DBG] pgmap v84: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:01 smithi006 bash[26022]: audit 2023-11-17T05:36:01.173947+0000 mon.smithi006 (mon.0) 459 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.235247+0000 mon.smithi006 (mon.0) 460 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-17T05:36:25.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: cluster 2023-11-17T05:36:01.235332+0000 mon.smithi006 (mon.0) 461 : cluster [DBG] osdmap e23: 8 total, 3 up, 8 in 2023-11-17T05:36:25.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.235556+0000 mon.smithi006 (mon.0) 462 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.236366+0000 mon.smithi006 (mon.0) 463 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.717 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.237001+0000 mon.smithi006 (mon.0) 464 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.717 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.237596+0000 mon.smithi006 (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.717 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.238295+0000 mon.smithi006 (mon.0) 466 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.238841+0000 mon.smithi006 (mon.0) 467 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.622430+0000 mon.smithi156 (mon.1) 14 : audit [INF] from='osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T05:36:25.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:02 smithi006 bash[26022]: audit 2023-11-17T05:36:01.623389+0000 mon.smithi006 (mon.0) 468 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T05:36:25.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.239903+0000 mon.smithi006 (mon.0) 469 : audit [INF] from='osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.240112+0000 mon.smithi006 (mon.0) 470 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.240274+0000 mon.smithi006 (mon.0) 471 : cluster [DBG] osdmap e24: 8 total, 3 up, 8 in 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.241289+0000 mon.smithi006 (mon.0) 472 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.241410+0000 mon.smithi156 (mon.1) 15 : audit [INF] from='osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.241956+0000 mon.smithi006 (mon.0) 473 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.243258+0000 mon.smithi006 (mon.0) 474 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.247998+0000 mon.smithi006 (mon.0) 475 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.248439+0000 mon.smithi006 (mon.0) 476 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.248768+0000 mon.smithi006 (mon.0) 477 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:02.249572+0000 mon.smithi006 (mon.0) 478 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.508643+0000 mgr.smithi006.odexej (mgr.14176) 126 : cluster [DBG] pgmap v87: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:03.242413+0000 mon.smithi006 (mon.0) 479 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: cluster 2023-11-17T05:36:03.242538+0000 mon.smithi006 (mon.0) 480 : cluster [INF] osd.3 [v2:172.21.15.6:6810/1794080715,v1:172.21.15.6:6811/1794080715] boot 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: cluster 2023-11-17T05:36:03.242618+0000 mon.smithi006 (mon.0) 481 : cluster [DBG] osdmap e25: 8 total, 4 up, 8 in 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:03 smithi006 bash[26022]: audit 2023-11-17T05:36:03.242818+0000 mon.smithi006 (mon.0) 482 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.214136+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.214259+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.244906+0000 mon.smithi006 (mon.0) 483 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.748 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.245239+0000 mon.smithi006 (mon.0) 484 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.750 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.245543+0000 mon.smithi006 (mon.0) 485 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.750 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.245838+0000 mon.smithi006 (mon.0) 486 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.770 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.251640+0000 mon.smithi006 (mon.0) 487 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.828 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:04 smithi006 bash[26022]: audit 2023-11-17T05:36:03.986050+0000 mon.smithi006 (mon.0) 488 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.655792+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:02.655887+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:04.245970+0000 mon.smithi006 (mon.0) 489 : cluster [INF] osd.4 [v2:172.21.15.156:6816/2624113389,v1:172.21.15.156:6817/2624113389] boot 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:04.246061+0000 mon.smithi006 (mon.0) 490 : cluster [DBG] osdmap e26: 8 total, 5 up, 8 in 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:04.246356+0000 mon.smithi006 (mon.0) 491 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:04.247683+0000 mon.smithi006 (mon.0) 492 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:04.247922+0000 mon.smithi006 (mon.0) 493 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:04.248203+0000 mon.smithi006 (mon.0) 494 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:04.508950+0000 mgr.smithi006.odexej (mgr.14176) 127 : cluster [DBG] pgmap v90: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-11-17T05:36:25.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.185265+0000 mon.smithi156 (mon.1) 16 : audit [INF] from='osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T05:36:25.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.186213+0000 mon.smithi006 (mon.0) 495 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T05:36:25.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.246841+0000 mon.smithi006 (mon.0) 496 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-17T05:36:25.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: cluster 2023-11-17T05:36:05.246960+0000 mon.smithi006 (mon.0) 497 : cluster [DBG] osdmap e27: 8 total, 5 up, 8 in 2023-11-17T05:36:25.932 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.247138+0000 mon.smithi006 (mon.0) 498 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.247487+0000 mon.smithi006 (mon.0) 499 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:05 smithi006 bash[26022]: audit 2023-11-17T05:36:05.247706+0000 mon.smithi006 (mon.0) 500 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:05.248912+0000 mon.smithi156 (mon.1) 17 : audit [INF] from='osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:05.249741+0000 mon.smithi006 (mon.0) 501 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.241174+0000 mon.smithi006 (mon.0) 502 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.249282+0000 mon.smithi006 (mon.0) 503 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.249384+0000 mon.smithi006 (mon.0) 504 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: cluster 2023-11-17T05:36:06.249447+0000 mon.smithi006 (mon.0) 505 : cluster [DBG] osdmap e28: 8 total, 5 up, 8 in 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.249614+0000 mon.smithi006 (mon.0) 506 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.250062+0000 mon.smithi006 (mon.0) 507 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.250329+0000 mon.smithi006 (mon.0) 508 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:25.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:06 smithi006 bash[26022]: audit 2023-11-17T05:36:06.250605+0000 mon.smithi006 (mon.0) 509 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:07 smithi006 bash[26022]: audit 2023-11-17T05:36:06.256896+0000 mon.smithi006 (mon.0) 510 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:07 smithi006 bash[26022]: cluster 2023-11-17T05:36:06.509474+0000 mgr.smithi006.odexej (mgr.14176) 128 : cluster [DBG] pgmap v93: 1 pgs: 1 remapped+peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: cluster 2023-11-17T05:36:06.205406+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: cluster 2023-11-17T05:36:06.205508+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.251910+0000 mon.smithi006 (mon.0) 511 : audit [INF] from='osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: cluster 2023-11-17T05:36:07.252072+0000 mon.smithi006 (mon.0) 512 : cluster [INF] osd.6 [v2:172.21.15.156:6824/2711251943,v1:172.21.15.156:6825/2711251943] boot 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: cluster 2023-11-17T05:36:07.252160+0000 mon.smithi006 (mon.0) 513 : cluster [DBG] osdmap e29: 8 total, 6 up, 8 in 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.252476+0000 mon.smithi006 (mon.0) 514 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.254188+0000 mon.smithi006 (mon.0) 515 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.257784+0000 mon.smithi006 (mon.0) 516 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.263455+0000 mon.smithi006 (mon.0) 517 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:08 smithi006 bash[26022]: audit 2023-11-17T05:36:07.549005+0000 mon.smithi006 (mon.0) 518 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: cluster 2023-11-17T05:36:07.269468+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: cluster 2023-11-17T05:36:07.269959+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: audit 2023-11-17T05:36:08.269663+0000 mon.smithi006 (mon.0) 519 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: cluster 2023-11-17T05:36:08.275843+0000 mon.smithi006 (mon.0) 520 : cluster [INF] osd.5 [v2:172.21.15.6:6818/1576530604,v1:172.21.15.6:6819/1576530604] boot 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: cluster 2023-11-17T05:36:08.275910+0000 mon.smithi006 (mon.0) 521 : cluster [DBG] osdmap e30: 8 total, 7 up, 8 in 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: audit 2023-11-17T05:36:08.276052+0000 mon.smithi006 (mon.0) 522 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: audit 2023-11-17T05:36:08.276521+0000 mon.smithi006 (mon.0) 523 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:09 smithi006 bash[26022]: cluster 2023-11-17T05:36:08.510013+0000 mgr.smithi006.odexej (mgr.14176) 129 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:10 smithi006 bash[26022]: cluster 2023-11-17T05:36:09.281399+0000 mon.smithi006 (mon.0) 524 : cluster [DBG] osdmap e31: 8 total, 7 up, 8 in 2023-11-17T05:36:25.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:10 smithi006 bash[26022]: audit 2023-11-17T05:36:09.281660+0000 mon.smithi006 (mon.0) 525 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:25.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:11 smithi006 bash[26022]: cluster 2023-11-17T05:36:10.510605+0000 mgr.smithi006.odexej (mgr.14176) 130 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:25.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:11 smithi006 bash[26022]: audit 2023-11-17T05:36:11.155988+0000 mon.smithi006 (mon.0) 526 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-17T05:36:25.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:13 smithi006 bash[26022]: audit 2023-11-17T05:36:12.153652+0000 mon.smithi006 (mon.0) 527 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-17T05:36:25.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:13 smithi006 bash[26022]: cluster 2023-11-17T05:36:12.153748+0000 mon.smithi006 (mon.0) 528 : cluster [DBG] osdmap e32: 8 total, 7 up, 8 in 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:13 smithi006 bash[26022]: audit 2023-11-17T05:36:12.153910+0000 mon.smithi006 (mon.0) 529 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:13 smithi006 bash[26022]: audit 2023-11-17T05:36:12.154478+0000 mon.smithi006 (mon.0) 530 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:13 smithi006 bash[26022]: cluster 2023-11-17T05:36:12.511114+0000 mgr.smithi006.odexej (mgr.14176) 131 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: cluster 2023-11-17T05:36:12.140134+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: cluster 2023-11-17T05:36:12.140219+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: audit 2023-11-17T05:36:13.156824+0000 mon.smithi006 (mon.0) 531 : audit [INF] from='osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2023-11-17T05:36:26.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: cluster 2023-11-17T05:36:13.156952+0000 mon.smithi006 (mon.0) 532 : cluster [DBG] osdmap e33: 8 total, 7 up, 8 in 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: audit 2023-11-17T05:36:13.157558+0000 mon.smithi006 (mon.0) 533 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:14 smithi006 bash[26022]: audit 2023-11-17T05:36:13.164765+0000 mon.smithi006 (mon.0) 534 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: audit 2023-11-17T05:36:14.169144+0000 mon.smithi006 (mon.0) 535 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: cluster 2023-11-17T05:36:14.179288+0000 mon.smithi006 (mon.0) 536 : cluster [INF] osd.7 [v2:172.21.15.6:6826/4133905079,v1:172.21.15.6:6827/4133905079] boot 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: cluster 2023-11-17T05:36:14.179456+0000 mon.smithi006 (mon.0) 537 : cluster [DBG] osdmap e34: 8 total, 8 up, 8 in 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: audit 2023-11-17T05:36:14.180128+0000 mon.smithi006 (mon.0) 538 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: audit 2023-11-17T05:36:14.207007+0000 mon.smithi006 (mon.0) 539 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: audit 2023-11-17T05:36:14.210058+0000 mon.smithi006 (mon.0) 540 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:15 smithi006 bash[26022]: cluster 2023-11-17T05:36:14.511752+0000 mgr.smithi006.odexej (mgr.14176) 132 : cluster [DBG] pgmap v103: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:16 smithi006 bash[26022]: cluster 2023-11-17T05:36:15.177779+0000 mon.smithi006 (mon.0) 541 : cluster [DBG] osdmap e35: 8 total, 8 up, 8 in 2023-11-17T05:36:26.024 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:17 smithi006 bash[26022]: cluster 2023-11-17T05:36:16.190716+0000 mon.smithi006 (mon.0) 542 : cluster [DBG] osdmap e36: 8 total, 8 up, 8 in 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:17 smithi006 bash[26022]: cluster 2023-11-17T05:36:16.512244+0000 mgr.smithi006.odexej (mgr.14176) 133 : cluster [DBG] pgmap v106: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:19 smithi006 bash[26022]: cluster 2023-11-17T05:36:18.512825+0000 mgr.smithi006.odexej (mgr.14176) 134 : cluster [DBG] pgmap v107: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:20 smithi006 bash[26022]: audit 2023-11-17T05:36:19.852134+0000 mon.smithi006 (mon.0) 543 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:20 smithi006 bash[26022]: audit 2023-11-17T05:36:20.499892+0000 mon.smithi006 (mon.0) 544 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:21 smithi006 bash[26022]: cluster 2023-11-17T05:36:20.513376+0000 mgr.smithi006.odexej (mgr.14176) 135 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:26.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:23 smithi006 bash[26022]: cluster 2023-11-17T05:36:22.513933+0000 mgr.smithi006.odexej (mgr.14176) 136 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:26.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T05:36:26.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:25 smithi006 bash[26022]: cluster 2023-11-17T05:36:24.514569+0000 mgr.smithi006.odexej (mgr.14176) 137 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:26.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:25 smithi156 bash[29655]: cluster 2023-11-17T05:36:24.514569+0000 mgr.smithi006.odexej (mgr.14176) 137 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:27.413 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:27 smithi006 bash[26022]: cluster 2023-11-17T05:36:26.515226+0000 mgr.smithi006.odexej (mgr.14176) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:27.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:27 smithi156 bash[29655]: cluster 2023-11-17T05:36:26.515226+0000 mgr.smithi006.odexej (mgr.14176) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:29.912 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:29 smithi006 bash[26022]: cluster 2023-11-17T05:36:28.515825+0000 mgr.smithi006.odexej (mgr.14176) 139 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:29.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:29 smithi006 bash[26022]: audit 2023-11-17T05:36:29.400585+0000 mon.smithi006 (mon.0) 545 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:29.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:29 smithi156 bash[29655]: cluster 2023-11-17T05:36:28.515825+0000 mgr.smithi006.odexej (mgr.14176) 139 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:29.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:29 smithi156 bash[29655]: audit 2023-11-17T05:36:29.400585+0000 mon.smithi006 (mon.0) 545 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:30.446 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:36:30.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: cephadm 2023-11-17T05:36:29.391096+0000 mgr.smithi006.odexej (mgr.14176) 140 : cephadm [INF] Detected new or changed devices on smithi156 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:29.685736+0000 mon.smithi006 (mon.0) 546 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:29.687743+0000 mon.smithi006 (mon.0) 547 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:29.689268+0000 mon.smithi006 (mon.0) 548 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:29.690751+0000 mon.smithi006 (mon.0) 549 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: cephadm 2023-11-17T05:36:29.691619+0000 mgr.smithi006.odexej (mgr.14176) 141 : cephadm [INF] Adjusting osd_memory_target on smithi156 to 4041M 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:29.696876+0000 mon.smithi006 (mon.0) 550 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:30 smithi006 bash[26022]: audit 2023-11-17T05:36:30.445905+0000 mon.smithi006 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.6:0/218866314' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:36:30.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: cephadm 2023-11-17T05:36:29.391096+0000 mgr.smithi006.odexej (mgr.14176) 140 : cephadm [INF] Detected new or changed devices on smithi156 2023-11-17T05:36:30.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:29.685736+0000 mon.smithi006 (mon.0) 546 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:29.687743+0000 mon.smithi006 (mon.0) 547 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:29.689268+0000 mon.smithi006 (mon.0) 548 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:29.690751+0000 mon.smithi006 (mon.0) 549 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:30.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: cephadm 2023-11-17T05:36:29.691619+0000 mgr.smithi006.odexej (mgr.14176) 141 : cephadm [INF] Adjusting osd_memory_target on smithi156 to 4041M 2023-11-17T05:36:30.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:29.696876+0000 mon.smithi006 (mon.0) 550 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:30.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:30 smithi156 bash[29655]: audit 2023-11-17T05:36:30.445905+0000 mon.smithi006 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.6:0/218866314' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T05:36:31.439 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"num_osds":8,"num_up_osds":8,"osd_up_since":1700199374,"num_in_osds":8,"osd_in_since":1700199330,"num_remapped_pgs":0} 2023-11-17T05:36:31.439 INFO:tasks.cephadm:Setting up client nodes... 2023-11-17T05:36:31.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-17T05:36:32.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:31 smithi006 bash[26022]: cluster 2023-11-17T05:36:30.516262+0000 mgr.smithi006.odexej (mgr.14176) 142 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:32.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:31 smithi156 bash[29655]: cluster 2023-11-17T05:36:30.516262+0000 mgr.smithi006.odexej (mgr.14176) 142 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: cephadm 2023-11-17T05:36:31.727335+0000 mgr.smithi006.odexej (mgr.14176) 143 : cephadm [INF] Detected new or changed devices on smithi006 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:31.738684+0000 mon.smithi006 (mon.0) 552 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.039871+0000 mon.smithi006 (mon.0) 553 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.041433+0000 mon.smithi006 (mon.0) 554 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.042962+0000 mon.smithi006 (mon.0) 555 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.044143+0000 mon.smithi006 (mon.0) 556 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.050116+0000 mon.smithi006 (mon.0) 557 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:32.825 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.052147+0000 mon.smithi006 (mon.0) 558 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.053247+0000 mon.smithi006 (mon.0) 559 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.060767+0000 mon.smithi006 (mon.0) 560 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.068256+0000 mon.smithi006 (mon.0) 561 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.073876+0000 mon.smithi006 (mon.0) 562 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.077418+0000 mon.smithi006 (mon.0) 563 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.077802+0000 mon.smithi006 (mon.0) 564 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:32.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:32 smithi006 bash[26022]: audit 2023-11-17T05:36:32.078844+0000 mon.smithi006 (mon.0) 565 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:33.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: cephadm 2023-11-17T05:36:31.727335+0000 mgr.smithi006.odexej (mgr.14176) 143 : cephadm [INF] Detected new or changed devices on smithi006 2023-11-17T05:36:33.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:31.738684+0000 mon.smithi006 (mon.0) 552 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.039871+0000 mon.smithi006 (mon.0) 553 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.041433+0000 mon.smithi006 (mon.0) 554 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.042962+0000 mon.smithi006 (mon.0) 555 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.044143+0000 mon.smithi006 (mon.0) 556 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.050116+0000 mon.smithi006 (mon.0) 557 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.052147+0000 mon.smithi006 (mon.0) 558 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.053247+0000 mon.smithi006 (mon.0) 559 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.060767+0000 mon.smithi006 (mon.0) 560 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.068256+0000 mon.smithi006 (mon.0) 561 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.073876+0000 mon.smithi006 (mon.0) 562 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.077418+0000 mon.smithi006 (mon.0) 563 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T05:36:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.077802+0000 mon.smithi006 (mon.0) 564 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:33.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:32 smithi156 bash[29655]: audit 2023-11-17T05:36:32.078844+0000 mon.smithi006 (mon.0) 565 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:36:33.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:33 smithi006 bash[26022]: cephadm 2023-11-17T05:36:32.044883+0000 mgr.smithi006.odexej (mgr.14176) 144 : cephadm [INF] Adjusting osd_memory_target on smithi006 to 3274M 2023-11-17T05:36:33.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:33 smithi006 bash[26022]: cluster 2023-11-17T05:36:32.516690+0000 mgr.smithi006.odexej (mgr.14176) 145 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:33.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:33 smithi156 bash[29655]: cephadm 2023-11-17T05:36:32.044883+0000 mgr.smithi006.odexej (mgr.14176) 144 : cephadm [INF] Adjusting osd_memory_target on smithi006 to 3274M 2023-11-17T05:36:33.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:33 smithi156 bash[29655]: cluster 2023-11-17T05:36:32.516690+0000 mgr.smithi006.odexej (mgr.14176) 145 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:34.709 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2023-11-17T05:36:34.709 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDi+1ZlGufeKRAAQPjAnkmamKCrSD28pOcqGw== 2023-11-17T05:36:35.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:34 smithi006 bash[26022]: audit 2023-11-17T05:36:34.702294+0000 mon.smithi006 (mon.0) 566 : audit [INF] from='client.? 172.21.15.6:0/839480839' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:35.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:34 smithi006 bash[26022]: audit 2023-11-17T05:36:34.708447+0000 mon.smithi006 (mon.0) 567 : audit [INF] from='client.? 172.21.15.6:0/839480839' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T05:36:35.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:34 smithi156 bash[29655]: audit 2023-11-17T05:36:34.702294+0000 mon.smithi006 (mon.0) 566 : audit [INF] from='client.? 172.21.15.6:0/839480839' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:35.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:34 smithi156 bash[29655]: audit 2023-11-17T05:36:34.708447+0000 mon.smithi006 (mon.0) 567 : audit [INF] from='client.? 172.21.15.6:0/839480839' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T05:36:35.358 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T05:36:35.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-17T05:36:35.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-17T05:36:35.382 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-17T05:36:36.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:35 smithi006 bash[26022]: cluster 2023-11-17T05:36:34.517147+0000 mgr.smithi006.odexej (mgr.14176) 146 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:36.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:35 smithi156 bash[29655]: cluster 2023-11-17T05:36:34.517147+0000 mgr.smithi006.odexej (mgr.14176) 146 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:37.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:37 smithi156 bash[29655]: cluster 2023-11-17T05:36:36.517735+0000 mgr.smithi006.odexej (mgr.14176) 147 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:37.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:37 smithi006 bash[26022]: cluster 2023-11-17T05:36:36.517735+0000 mgr.smithi006.odexej (mgr.14176) 147 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:38.987 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2023-11-17T05:36:38.987 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDm+1ZlKk5YOhAA6jNQlxgKVBjjeNOOKv4taw== 2023-11-17T05:36:39.837 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:39 smithi156 bash[29655]: cluster 2023-11-17T05:36:38.518348+0000 mgr.smithi006.odexej (mgr.14176) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:39.838 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:39 smithi156 bash[29655]: audit 2023-11-17T05:36:38.977611+0000 mon.smithi156 (mon.1) 18 : audit [INF] from='client.? 172.21.15.156:0/722908442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:39.838 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:39 smithi156 bash[29655]: audit 2023-11-17T05:36:38.978504+0000 mon.smithi006 (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:39.838 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:39 smithi156 bash[29655]: audit 2023-11-17T05:36:38.986020+0000 mon.smithi006 (mon.0) 569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T05:36:39.839 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T05:36:39.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-11-17T05:36:39.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-17T05:36:39.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T05:36:39.876 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:39 smithi006 bash[26022]: cluster 2023-11-17T05:36:38.518348+0000 mgr.smithi006.odexej (mgr.14176) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:39.876 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:39 smithi006 bash[26022]: audit 2023-11-17T05:36:38.977611+0000 mon.smithi156 (mon.1) 18 : audit [INF] from='client.? 172.21.15.156:0/722908442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:39.877 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:39 smithi006 bash[26022]: audit 2023-11-17T05:36:38.978504+0000 mon.smithi006 (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T05:36:39.877 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:39 smithi006 bash[26022]: audit 2023-11-17T05:36:38.986020+0000 mon.smithi006 (mon.0) 569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T05:36:41.912 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:41 smithi006 bash[26022]: cluster 2023-11-17T05:36:40.519016+0000 mgr.smithi006.odexej (mgr.14176) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:41.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:41 smithi156 bash[29655]: cluster 2023-11-17T05:36:40.519016+0000 mgr.smithi006.odexej (mgr.14176) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:43.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:43 smithi006 bash[26022]: cluster 2023-11-17T05:36:42.519686+0000 mgr.smithi006.odexej (mgr.14176) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:43.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:43 smithi156 bash[29655]: cluster 2023-11-17T05:36:42.519686+0000 mgr.smithi006.odexej (mgr.14176) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:44.708 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:36:45.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:45 smithi006 bash[26022]: cluster 2023-11-17T05:36:44.520274+0000 mgr.smithi006.odexej (mgr.14176) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:45.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:45 smithi156 bash[29655]: cluster 2023-11-17T05:36:44.520274+0000 mgr.smithi006.odexej (mgr.14176) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:47.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:47 smithi006 bash[26022]: cluster 2023-11-17T05:36:46.520935+0000 mgr.smithi006.odexej (mgr.14176) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:47.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:47 smithi156 bash[29655]: cluster 2023-11-17T05:36:46.520935+0000 mgr.smithi006.odexej (mgr.14176) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:48.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:36:48.466 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","created":"2023-11-17T05:31:35.035374+0000","modified":"2023-11-17T05:36:16.175038+0000","last_up_change":"2023-11-17T05:36:14.160485+0000","last_in_change":"2023-11-17T05:35:30.806755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T05:35:58.676312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bd001464-a43a-4e8e-b89f-737e2fb6afcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6801","nonce":3889302703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6803","nonce":3889302703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6807","nonce":3889302703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6805","nonce":3889302703}]},"public_addr":"172.21.15.156:6801/3889302703","cluster_addr":"172.21.15.156:6803/3889302703","heartbeat_back_addr":"172.21.15.156:6807/3889302703","heartbeat_front_addr":"172.21.15.156:6805/3889302703","state":["exists","up"]},{"osd":1,"uuid":"eefe282d-2c80-4ba7-9d5b-5352b05ae877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6803","nonce":3517964296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6805","nonce":3517964296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6809","nonce":3517964296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6807","nonce":3517964296}]},"public_addr":"172.21.15.6:6803/3517964296","cluster_addr":"172.21.15.6:6805/3517964296","heartbeat_back_addr":"172.21.15.6:6809/3517964296","heartbeat_front_addr":"172.21.15.6:6807/3517964296","state":["exists","up"]},{"osd":2,"uuid":"49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6809","nonce":533925933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6811","nonce":533925933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6815","nonce":533925933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6813","nonce":533925933}]},"public_addr":"172.21.15.156:6809/533925933","cluster_addr":"172.21.15.156:6811/533925933","heartbeat_back_addr":"172.21.15.156:6815/533925933","heartbeat_front_addr":"172.21.15.156:6813/533925933","state":["exists","up"]},{"osd":3,"uuid":"96fca6d4-495b-4520-b8be-df976c05aa30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6811","nonce":1794080715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6813","nonce":1794080715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6817","nonce":1794080715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6815","nonce":1794080715}]},"public_addr":"172.21.15.6:6811/1794080715","cluster_addr":"172.21.15.6:6813/1794080715","heartbeat_back_addr":"172.21.15.6:6817/1794080715","heartbeat_front_addr":"172.21.15.6:6815/1794080715","state":["exists","up"]},{"osd":4,"uuid":"5866a83c-d866-4dcf-868c-7150a44f314b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6817","nonce":2624113389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6819","nonce":2624113389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6823","nonce":2624113389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6821","nonce":2624113389}]},"public_addr":"172.21.15.156:6817/2624113389","cluster_addr":"172.21.15.156:6819/2624113389","heartbeat_back_addr":"172.21.15.156:6823/2624113389","heartbeat_front_addr":"172.21.15.156:6821/2624113389","state":["exists","up"]},{"osd":5,"uuid":"2e3f232f-34c0-4619-b611-c288652cf48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6819","nonce":1576530604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6821","nonce":1576530604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6825","nonce":1576530604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6823","nonce":1576530604}]},"public_addr":"172.21.15.6:6819/1576530604","cluster_addr":"172.21.15.6:6821/1576530604","heartbeat_back_addr":"172.21.15.6:6825/1576530604","heartbeat_front_addr":"172.21.15.6:6823/1576530604","state":["exists","up"]},{"osd":6,"uuid":"ab650abe-2841-4f3b-93e6-76be0832bfb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6825","nonce":2711251943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6827","nonce":2711251943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6831","nonce":2711251943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6829","nonce":2711251943}]},"public_addr":"172.21.15.156:6825/2711251943","cluster_addr":"172.21.15.156:6827/2711251943","heartbeat_back_addr":"172.21.15.156:6831/2711251943","heartbeat_front_addr":"172.21.15.156:6829/2711251943","state":["exists","up"]},{"osd":7,"uuid":"80c4e482-5a64-46d6-9ad0-48b038823b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6827","nonce":4133905079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6829","nonce":4133905079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6833","nonce":4133905079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6831","nonce":4133905079}]},"public_addr":"172.21.15.6:6827/4133905079","cluster_addr":"172.21.15.6:6829/4133905079","heartbeat_back_addr":"172.21.15.6:6833/4133905079","heartbeat_front_addr":"172.21.15.6:6831/4133905079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:52.485501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.905548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.947047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.214268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.655893+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:07.269964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:06.205514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:12.140224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6800/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/1056254422":"2023-11-18T05:33:34.469250+0000","172.21.15.6:6801/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/4185379340":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/231566654":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6801/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:0/1766155535":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:6801/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/3021497559":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/4020642765":"2023-11-18T05:32:02.657299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T05:36:48.881 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:48 smithi156 bash[29655]: audit 2023-11-17T05:36:48.465032+0000 mon.smithi006 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.6:0/3263665386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:36:48.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:48 smithi006 bash[26022]: audit 2023-11-17T05:36:48.465032+0000 mon.smithi006 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.6:0/3263665386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:36:49.246 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-11-17T05:35:58.676312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2023-11-17T05:36:49.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd pool get device_health_metrics pg_num 2023-11-17T05:36:49.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:49 smithi156 bash[29655]: cluster 2023-11-17T05:36:48.521544+0000 mgr.smithi006.odexej (mgr.14176) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:49.792 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:49 smithi006 bash[26022]: cluster 2023-11-17T05:36:48.521544+0000 mgr.smithi006.odexej (mgr.14176) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:51.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:51 smithi006 bash[26022]: cluster 2023-11-17T05:36:50.522203+0000 mgr.smithi006.odexej (mgr.14176) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:51.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:51 smithi156 bash[29655]: cluster 2023-11-17T05:36:50.522203+0000 mgr.smithi006.odexej (mgr.14176) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:53.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:53 smithi006 bash[26022]: cluster 2023-11-17T05:36:52.522842+0000 mgr.smithi006.odexej (mgr.14176) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:53.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:53 smithi156 bash[29655]: cluster 2023-11-17T05:36:52.522842+0000 mgr.smithi006.odexej (mgr.14176) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:54.146 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:36:55.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:55 smithi006 bash[26022]: cluster 2023-11-17T05:36:54.523553+0000 mgr.smithi006.odexej (mgr.14176) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:55.890 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:55 smithi156 bash[29655]: cluster 2023-11-17T05:36:54.523553+0000 mgr.smithi006.odexej (mgr.14176) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:57.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:57 smithi156 bash[29655]: cluster 2023-11-17T05:36:56.524421+0000 mgr.smithi006.odexej (mgr.14176) 157 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:57.576 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:57 smithi006 bash[26022]: cluster 2023-11-17T05:36:56.524421+0000 mgr.smithi006.odexej (mgr.14176) 157 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:58.162 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2023-11-17T05:36:58.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:58 smithi156 bash[29655]: audit 2023-11-17T05:36:58.161299+0000 mon.smithi006 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.6:0/2850448359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-11-17T05:36:58.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:58 smithi006 bash[26022]: audit 2023-11-17T05:36:58.161299+0000 mon.smithi006 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.6:0/2850448359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-11-17T05:36:58.906 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-17T05:36:58.907 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-17T05:36:58.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-17T05:36:59.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:59 smithi006 bash[26022]: cluster 2023-11-17T05:36:58.525104+0000 mgr.smithi006.odexej (mgr.14176) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:59.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:36:59 smithi006 bash[26022]: audit 2023-11-17T05:36:58.588135+0000 mon.smithi006 (mon.0) 572 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:36:59.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:59 smithi156 bash[29655]: cluster 2023-11-17T05:36:58.525104+0000 mgr.smithi006.odexej (mgr.14176) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:36:59.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:36:59 smithi156 bash[29655]: audit 2023-11-17T05:36:58.588135+0000 mon.smithi006 (mon.0) 572 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:37:01.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:01 smithi006 bash[26022]: audit 2023-11-17T05:37:00.233409+0000 mon.smithi006 (mon.0) 573 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:37:01.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:01 smithi006 bash[26022]: cluster 2023-11-17T05:37:00.525767+0000 mgr.smithi006.odexej (mgr.14176) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:01.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:01 smithi156 bash[29655]: audit 2023-11-17T05:37:00.233409+0000 mon.smithi006 (mon.0) 573 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:37:01.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:01 smithi156 bash[29655]: cluster 2023-11-17T05:37:00.525767+0000 mgr.smithi006.odexej (mgr.14176) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:03.796 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:03.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:03 smithi006 bash[26022]: cluster 2023-11-17T05:37:02.526464+0000 mgr.smithi006.odexej (mgr.14176) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:03.929 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:03 smithi156 bash[29655]: cluster 2023-11-17T05:37:02.526464+0000 mgr.smithi006.odexej (mgr.14176) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:05.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:05 smithi006 bash[26022]: cluster 2023-11-17T05:37:04.527142+0000 mgr.smithi006.odexej (mgr.14176) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:05.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:05 smithi156 bash[29655]: cluster 2023-11-17T05:37:04.527142+0000 mgr.smithi006.odexej (mgr.14176) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:06.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:37:07.345 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"active_gid":14176,"active_name":"smithi006.odexej","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1796889019},{"type":"v1","addr":"172.21.15.6:6801","nonce":1796889019}]},"active_addr":"172.21.15.6:6801/1796889019","active_change":"2023-11-17T05:33:34.469395+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":14198,"name":"smithi156.fphsnp","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.6:8443/","prometheus":"http://172.21.15.6:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":4,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":4272217230}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3482813200}]}]} 2023-11-17T05:37:07.348 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-17T05:37:07.348 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-17T05:37:07.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T05:37:07.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:07 smithi156 bash[29655]: cluster 2023-11-17T05:37:06.527771+0000 mgr.smithi006.odexej (mgr.14176) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:07.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:07 smithi156 bash[29655]: audit 2023-11-17T05:37:06.747137+0000 mon.smithi006 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.6:0/2562407247' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-17T05:37:07.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:07 smithi006 bash[26022]: cluster 2023-11-17T05:37:06.527771+0000 mgr.smithi006.odexej (mgr.14176) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:07.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:07 smithi006 bash[26022]: audit 2023-11-17T05:37:06.747137+0000 mon.smithi006 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.6:0/2562407247' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-17T05:37:09.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:09 smithi006 bash[26022]: cluster 2023-11-17T05:37:08.528385+0000 mgr.smithi006.odexej (mgr.14176) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:09.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:09 smithi156 bash[29655]: cluster 2023-11-17T05:37:08.528385+0000 mgr.smithi006.odexej (mgr.14176) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:11.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:11 smithi006 bash[26022]: cluster 2023-11-17T05:37:10.529016+0000 mgr.smithi006.odexej (mgr.14176) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:11.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:11 smithi156 bash[29655]: cluster 2023-11-17T05:37:10.529016+0000 mgr.smithi006.odexej (mgr.14176) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:12.201 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:13.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:13 smithi006 bash[26022]: cluster 2023-11-17T05:37:12.529608+0000 mgr.smithi006.odexej (mgr.14176) 165 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:13.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:13 smithi156 bash[29655]: cluster 2023-11-17T05:37:12.529608+0000 mgr.smithi006.odexej (mgr.14176) 165 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:15.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:37:15.204 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","created":"2023-11-17T05:31:35.035374+0000","modified":"2023-11-17T05:36:16.175038+0000","last_up_change":"2023-11-17T05:36:14.160485+0000","last_in_change":"2023-11-17T05:35:30.806755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T05:35:58.676312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bd001464-a43a-4e8e-b89f-737e2fb6afcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6801","nonce":3889302703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6803","nonce":3889302703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6807","nonce":3889302703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6805","nonce":3889302703}]},"public_addr":"172.21.15.156:6801/3889302703","cluster_addr":"172.21.15.156:6803/3889302703","heartbeat_back_addr":"172.21.15.156:6807/3889302703","heartbeat_front_addr":"172.21.15.156:6805/3889302703","state":["exists","up"]},{"osd":1,"uuid":"eefe282d-2c80-4ba7-9d5b-5352b05ae877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6803","nonce":3517964296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6805","nonce":3517964296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6809","nonce":3517964296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6807","nonce":3517964296}]},"public_addr":"172.21.15.6:6803/3517964296","cluster_addr":"172.21.15.6:6805/3517964296","heartbeat_back_addr":"172.21.15.6:6809/3517964296","heartbeat_front_addr":"172.21.15.6:6807/3517964296","state":["exists","up"]},{"osd":2,"uuid":"49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6809","nonce":533925933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6811","nonce":533925933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6815","nonce":533925933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6813","nonce":533925933}]},"public_addr":"172.21.15.156:6809/533925933","cluster_addr":"172.21.15.156:6811/533925933","heartbeat_back_addr":"172.21.15.156:6815/533925933","heartbeat_front_addr":"172.21.15.156:6813/533925933","state":["exists","up"]},{"osd":3,"uuid":"96fca6d4-495b-4520-b8be-df976c05aa30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6811","nonce":1794080715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6813","nonce":1794080715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6817","nonce":1794080715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6815","nonce":1794080715}]},"public_addr":"172.21.15.6:6811/1794080715","cluster_addr":"172.21.15.6:6813/1794080715","heartbeat_back_addr":"172.21.15.6:6817/1794080715","heartbeat_front_addr":"172.21.15.6:6815/1794080715","state":["exists","up"]},{"osd":4,"uuid":"5866a83c-d866-4dcf-868c-7150a44f314b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6817","nonce":2624113389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6819","nonce":2624113389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6823","nonce":2624113389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6821","nonce":2624113389}]},"public_addr":"172.21.15.156:6817/2624113389","cluster_addr":"172.21.15.156:6819/2624113389","heartbeat_back_addr":"172.21.15.156:6823/2624113389","heartbeat_front_addr":"172.21.15.156:6821/2624113389","state":["exists","up"]},{"osd":5,"uuid":"2e3f232f-34c0-4619-b611-c288652cf48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6819","nonce":1576530604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6821","nonce":1576530604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6825","nonce":1576530604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6823","nonce":1576530604}]},"public_addr":"172.21.15.6:6819/1576530604","cluster_addr":"172.21.15.6:6821/1576530604","heartbeat_back_addr":"172.21.15.6:6825/1576530604","heartbeat_front_addr":"172.21.15.6:6823/1576530604","state":["exists","up"]},{"osd":6,"uuid":"ab650abe-2841-4f3b-93e6-76be0832bfb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6825","nonce":2711251943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6827","nonce":2711251943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6831","nonce":2711251943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6829","nonce":2711251943}]},"public_addr":"172.21.15.156:6825/2711251943","cluster_addr":"172.21.15.156:6827/2711251943","heartbeat_back_addr":"172.21.15.156:6831/2711251943","heartbeat_front_addr":"172.21.15.156:6829/2711251943","state":["exists","up"]},{"osd":7,"uuid":"80c4e482-5a64-46d6-9ad0-48b038823b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6827","nonce":4133905079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6829","nonce":4133905079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6833","nonce":4133905079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6831","nonce":4133905079}]},"public_addr":"172.21.15.6:6827/4133905079","cluster_addr":"172.21.15.6:6829/4133905079","heartbeat_back_addr":"172.21.15.6:6833/4133905079","heartbeat_front_addr":"172.21.15.6:6831/4133905079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:52.485501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.905548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.947047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.214268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.655893+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:07.269964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:06.205514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:12.140224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6800/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/1056254422":"2023-11-18T05:33:34.469250+0000","172.21.15.6:6801/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/4185379340":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/231566654":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6801/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:0/1766155535":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:6801/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/3021497559":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/4020642765":"2023-11-18T05:32:02.657299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T05:37:15.745 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-17T05:37:15.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T05:37:15.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:15 smithi006 bash[26022]: cluster 2023-11-17T05:37:14.530188+0000 mgr.smithi006.odexej (mgr.14176) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:15.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:15 smithi006 bash[26022]: audit 2023-11-17T05:37:15.201666+0000 mon.smithi006 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.6:0/3633718364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:37:15.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:15 smithi156 bash[29655]: cluster 2023-11-17T05:37:14.530188+0000 mgr.smithi006.odexej (mgr.14176) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:15.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:15 smithi156 bash[29655]: audit 2023-11-17T05:37:15.201666+0000 mon.smithi006 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.6:0/3633718364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:37:17.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:17 smithi006 bash[26022]: cluster 2023-11-17T05:37:16.530826+0000 mgr.smithi006.odexej (mgr.14176) 167 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:17.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:17 smithi156 bash[29655]: cluster 2023-11-17T05:37:16.530826+0000 mgr.smithi006.odexej (mgr.14176) 167 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:19.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:19 smithi006 bash[26022]: cluster 2023-11-17T05:37:18.531450+0000 mgr.smithi006.odexej (mgr.14176) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:19.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:19 smithi156 bash[29655]: cluster 2023-11-17T05:37:18.531450+0000 mgr.smithi006.odexej (mgr.14176) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:20.591 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:21.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:21 smithi006 bash[26022]: cluster 2023-11-17T05:37:20.532174+0000 mgr.smithi006.odexej (mgr.14176) 169 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:21.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:21 smithi156 bash[29655]: cluster 2023-11-17T05:37:20.532174+0000 mgr.smithi006.odexej (mgr.14176) 169 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:23.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:37:23.413 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"fsid":"7d14d910-850a-11ee-8dbe-212e2dc638e7","created":"2023-11-17T05:31:35.035374+0000","modified":"2023-11-17T05:36:16.175038+0000","last_up_change":"2023-11-17T05:36:14.160485+0000","last_in_change":"2023-11-17T05:35:30.806755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T05:35:58.676312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bd001464-a43a-4e8e-b89f-737e2fb6afcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6801","nonce":3889302703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6803","nonce":3889302703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6807","nonce":3889302703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3889302703},{"type":"v1","addr":"172.21.15.156:6805","nonce":3889302703}]},"public_addr":"172.21.15.156:6801/3889302703","cluster_addr":"172.21.15.156:6803/3889302703","heartbeat_back_addr":"172.21.15.156:6807/3889302703","heartbeat_front_addr":"172.21.15.156:6805/3889302703","state":["exists","up"]},{"osd":1,"uuid":"eefe282d-2c80-4ba7-9d5b-5352b05ae877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6803","nonce":3517964296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6805","nonce":3517964296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6809","nonce":3517964296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3517964296},{"type":"v1","addr":"172.21.15.6:6807","nonce":3517964296}]},"public_addr":"172.21.15.6:6803/3517964296","cluster_addr":"172.21.15.6:6805/3517964296","heartbeat_back_addr":"172.21.15.6:6809/3517964296","heartbeat_front_addr":"172.21.15.6:6807/3517964296","state":["exists","up"]},{"osd":2,"uuid":"49d6fd1e-8d83-4ffc-a61a-f5b85aa32be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6809","nonce":533925933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6811","nonce":533925933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6815","nonce":533925933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":533925933},{"type":"v1","addr":"172.21.15.156:6813","nonce":533925933}]},"public_addr":"172.21.15.156:6809/533925933","cluster_addr":"172.21.15.156:6811/533925933","heartbeat_back_addr":"172.21.15.156:6815/533925933","heartbeat_front_addr":"172.21.15.156:6813/533925933","state":["exists","up"]},{"osd":3,"uuid":"96fca6d4-495b-4520-b8be-df976c05aa30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6811","nonce":1794080715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6813","nonce":1794080715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6817","nonce":1794080715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1794080715},{"type":"v1","addr":"172.21.15.6:6815","nonce":1794080715}]},"public_addr":"172.21.15.6:6811/1794080715","cluster_addr":"172.21.15.6:6813/1794080715","heartbeat_back_addr":"172.21.15.6:6817/1794080715","heartbeat_front_addr":"172.21.15.6:6815/1794080715","state":["exists","up"]},{"osd":4,"uuid":"5866a83c-d866-4dcf-868c-7150a44f314b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6817","nonce":2624113389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6819","nonce":2624113389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6823","nonce":2624113389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2624113389},{"type":"v1","addr":"172.21.15.156:6821","nonce":2624113389}]},"public_addr":"172.21.15.156:6817/2624113389","cluster_addr":"172.21.15.156:6819/2624113389","heartbeat_back_addr":"172.21.15.156:6823/2624113389","heartbeat_front_addr":"172.21.15.156:6821/2624113389","state":["exists","up"]},{"osd":5,"uuid":"2e3f232f-34c0-4619-b611-c288652cf48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6819","nonce":1576530604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6821","nonce":1576530604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6825","nonce":1576530604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1576530604},{"type":"v1","addr":"172.21.15.6:6823","nonce":1576530604}]},"public_addr":"172.21.15.6:6819/1576530604","cluster_addr":"172.21.15.6:6821/1576530604","heartbeat_back_addr":"172.21.15.6:6825/1576530604","heartbeat_front_addr":"172.21.15.6:6823/1576530604","state":["exists","up"]},{"osd":6,"uuid":"ab650abe-2841-4f3b-93e6-76be0832bfb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6825","nonce":2711251943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6827","nonce":2711251943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6831","nonce":2711251943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2711251943},{"type":"v1","addr":"172.21.15.156:6829","nonce":2711251943}]},"public_addr":"172.21.15.156:6825/2711251943","cluster_addr":"172.21.15.156:6827/2711251943","heartbeat_back_addr":"172.21.15.156:6831/2711251943","heartbeat_front_addr":"172.21.15.156:6829/2711251943","state":["exists","up"]},{"osd":7,"uuid":"80c4e482-5a64-46d6-9ad0-48b038823b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6827","nonce":4133905079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6829","nonce":4133905079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6833","nonce":4133905079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":4133905079},{"type":"v1","addr":"172.21.15.6:6831","nonce":4133905079}]},"public_addr":"172.21.15.6:6827/4133905079","cluster_addr":"172.21.15.6:6829/4133905079","heartbeat_back_addr":"172.21.15.6:6833/4133905079","heartbeat_front_addr":"172.21.15.6:6831/4133905079","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:52.485501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.905548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:35:56.947047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.214268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:02.655893+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:07.269964+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:06.205514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T05:36:12.140224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6800/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/1056254422":"2023-11-18T05:33:34.469250+0000","172.21.15.6:6801/3193677044":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/4185379340":"2023-11-18T05:33:34.469250+0000","172.21.15.6:0/231566654":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6801/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/262320788":"2023-11-18T05:32:41.041678+0000","172.21.15.6:0/1766155535":"2023-11-18T05:32:41.041678+0000","172.21.15.6:6800/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:6801/22207928":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/3021497559":"2023-11-18T05:32:02.657299+0000","172.21.15.6:0/4020642765":"2023-11-18T05:32:02.657299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T05:37:23.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:23 smithi006 bash[26022]: cluster 2023-11-17T05:37:22.532929+0000 mgr.smithi006.odexej (mgr.14176) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:23.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:23 smithi006 bash[26022]: audit 2023-11-17T05:37:23.410306+0000 mon.smithi006 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.6:0/903865579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:37:23.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:23 smithi156 bash[29655]: cluster 2023-11-17T05:37:22.532929+0000 mgr.smithi006.odexej (mgr.14176) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:23.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:23 smithi156 bash[29655]: audit 2023-11-17T05:37:23.410306+0000 mon.smithi006 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.6:0/903865579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T05:37:23.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-17T05:37:25.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:25 smithi006 bash[26022]: cluster 2023-11-17T05:37:24.533571+0000 mgr.smithi006.odexej (mgr.14176) 171 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:25.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:25 smithi156 bash[29655]: cluster 2023-11-17T05:37:24.533571+0000 mgr.smithi006.odexej (mgr.14176) 171 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:27.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:27 smithi006 bash[26022]: cluster 2023-11-17T05:37:26.534206+0000 mgr.smithi006.odexej (mgr.14176) 172 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:27.715 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:27 smithi156 bash[29655]: cluster 2023-11-17T05:37:26.534206+0000 mgr.smithi006.odexej (mgr.14176) 172 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:29.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:29 smithi006 bash[26022]: cluster 2023-11-17T05:37:28.534927+0000 mgr.smithi006.odexej (mgr.14176) 173 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:29.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:29 smithi156 bash[29655]: cluster 2023-11-17T05:37:28.534927+0000 mgr.smithi006.odexej (mgr.14176) 173 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:31.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:31 smithi006 bash[26022]: cluster 2023-11-17T05:37:30.535566+0000 mgr.smithi006.odexej (mgr.14176) 174 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:31.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:31 smithi156 bash[29655]: cluster 2023-11-17T05:37:30.535566+0000 mgr.smithi006.odexej (mgr.14176) 174 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:32.136 INFO:teuthology.orchestra.run.smithi006.stdout:68719476758 2023-11-17T05:37:32.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-17T05:37:33.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:33 smithi006 bash[26022]: cluster 2023-11-17T05:37:32.536249+0000 mgr.smithi006.odexej (mgr.14176) 175 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:33.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:33 smithi156 bash[29655]: cluster 2023-11-17T05:37:32.536249+0000 mgr.smithi006.odexej (mgr.14176) 175 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:35.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:35 smithi006 bash[26022]: cluster 2023-11-17T05:37:34.536878+0000 mgr.smithi006.odexej (mgr.14176) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:35.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:35 smithi156 bash[29655]: cluster 2023-11-17T05:37:34.536878+0000 mgr.smithi006.odexej (mgr.14176) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:36.987 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:37.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:37 smithi006 bash[26022]: cluster 2023-11-17T05:37:36.537458+0000 mgr.smithi006.odexej (mgr.14176) 177 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:37.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:37 smithi156 bash[29655]: cluster 2023-11-17T05:37:36.537458+0000 mgr.smithi006.odexej (mgr.14176) 177 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:39.768 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:39 smithi006 bash[26022]: cluster 2023-11-17T05:37:38.538087+0000 mgr.smithi006.odexej (mgr.14176) 178 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:39.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:39 smithi156 bash[29655]: cluster 2023-11-17T05:37:38.538087+0000 mgr.smithi006.odexej (mgr.14176) 178 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:40.322 INFO:teuthology.orchestra.run.smithi006.stdout:85899345944 2023-11-17T05:37:40.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-17T05:37:41.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:41 smithi006 bash[26022]: cluster 2023-11-17T05:37:40.538655+0000 mgr.smithi006.odexej (mgr.14176) 179 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:41.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:41 smithi156 bash[29655]: cluster 2023-11-17T05:37:40.538655+0000 mgr.smithi006.odexej (mgr.14176) 179 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:43.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:43 smithi006 bash[26022]: cluster 2023-11-17T05:37:42.539211+0000 mgr.smithi006.odexej (mgr.14176) 180 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:43.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:43 smithi156 bash[29655]: cluster 2023-11-17T05:37:42.539211+0000 mgr.smithi006.odexej (mgr.14176) 180 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:45.171 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:45.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:45 smithi006 bash[26022]: cluster 2023-11-17T05:37:44.539819+0000 mgr.smithi006.odexej (mgr.14176) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:45.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:45 smithi156 bash[29655]: cluster 2023-11-17T05:37:44.539819+0000 mgr.smithi006.odexej (mgr.14176) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:47.478 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:47 smithi006 bash[26022]: cluster 2023-11-17T05:37:46.540378+0000 mgr.smithi006.odexej (mgr.14176) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:47.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:47 smithi156 bash[29655]: cluster 2023-11-17T05:37:46.540378+0000 mgr.smithi006.odexej (mgr.14176) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:48.315 INFO:teuthology.orchestra.run.smithi006.stdout:85899345945 2023-11-17T05:37:48.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-17T05:37:49.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:49 smithi006 bash[26022]: cluster 2023-11-17T05:37:48.540835+0000 mgr.smithi006.odexej (mgr.14176) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:49.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:49 smithi156 bash[29655]: cluster 2023-11-17T05:37:48.540835+0000 mgr.smithi006.odexej (mgr.14176) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:51.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:51 smithi006 bash[26022]: cluster 2023-11-17T05:37:50.541517+0000 mgr.smithi006.odexej (mgr.14176) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:51.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:51 smithi156 bash[29655]: cluster 2023-11-17T05:37:50.541517+0000 mgr.smithi006.odexej (mgr.14176) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:53.175 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:37:53.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:53 smithi006 bash[26022]: cluster 2023-11-17T05:37:52.542142+0000 mgr.smithi006.odexej (mgr.14176) 185 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:53.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:53 smithi156 bash[29655]: cluster 2023-11-17T05:37:52.542142+0000 mgr.smithi006.odexej (mgr.14176) 185 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:55.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:55 smithi006 bash[26022]: cluster 2023-11-17T05:37:54.542730+0000 mgr.smithi006.odexej (mgr.14176) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:55.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:55 smithi156 bash[29655]: cluster 2023-11-17T05:37:54.542730+0000 mgr.smithi006.odexej (mgr.14176) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:56.471 INFO:teuthology.orchestra.run.smithi006.stdout:107374182425 2023-11-17T05:37:56.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-17T05:37:57.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:57 smithi006 bash[26022]: cluster 2023-11-17T05:37:56.543315+0000 mgr.smithi006.odexej (mgr.14176) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:57.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:57 smithi156 bash[29655]: cluster 2023-11-17T05:37:56.543315+0000 mgr.smithi006.odexej (mgr.14176) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:59.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:37:59 smithi006 bash[26022]: cluster 2023-11-17T05:37:58.543896+0000 mgr.smithi006.odexej (mgr.14176) 188 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:37:59.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:37:59 smithi156 bash[29655]: cluster 2023-11-17T05:37:58.543896+0000 mgr.smithi006.odexej (mgr.14176) 188 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:00.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:00 smithi006 bash[26022]: audit 2023-11-17T05:38:00.238095+0000 mon.smithi006 (mon.0) 577 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:38:00.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:00 smithi156 bash[29655]: audit 2023-11-17T05:38:00.238095+0000 mon.smithi006 (mon.0) 577 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:38:01.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:01 smithi006 bash[26022]: cluster 2023-11-17T05:38:00.544481+0000 mgr.smithi006.odexej (mgr.14176) 189 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:01.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:01 smithi156 bash[29655]: cluster 2023-11-17T05:38:00.544481+0000 mgr.smithi006.odexej (mgr.14176) 189 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:03.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:03 smithi006 bash[26022]: cluster 2023-11-17T05:38:02.545048+0000 mgr.smithi006.odexej (mgr.14176) 190 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:03.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:03 smithi156 bash[29655]: cluster 2023-11-17T05:38:02.545048+0000 mgr.smithi006.odexej (mgr.14176) 190 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:04.534 INFO:teuthology.orchestra.run.smithi006.stdout:111669149721 2023-11-17T05:38:04.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-17T05:38:05.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:05 smithi006 bash[26022]: cluster 2023-11-17T05:38:04.545631+0000 mgr.smithi006.odexej (mgr.14176) 191 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:05.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:05 smithi156 bash[29655]: cluster 2023-11-17T05:38:04.545631+0000 mgr.smithi006.odexej (mgr.14176) 191 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:07.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:05.873215+0000 mon.smithi006 (mon.0) 578 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:06.392580+0000 mon.smithi006 (mon.0) 579 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:06.690694+0000 mon.smithi006 (mon.0) 580 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:38:07.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:06.692032+0000 mon.smithi006 (mon.0) 581 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:38:07.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:06.702768+0000 mon.smithi006 (mon.0) 582 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:06 smithi006 bash[26022]: audit 2023-11-17T05:38:06.708246+0000 mon.smithi006 (mon.0) 583 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:38:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:05.873215+0000 mon.smithi006 (mon.0) 578 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:06.392580+0000 mon.smithi006 (mon.0) 579 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:06.690694+0000 mon.smithi006 (mon.0) 580 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:38:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:06.692032+0000 mon.smithi006 (mon.0) 581 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:38:07.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:06.702768+0000 mon.smithi006 (mon.0) 582 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:38:07.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:06 smithi156 bash[29655]: audit 2023-11-17T05:38:06.708246+0000 mon.smithi006 (mon.0) 583 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:38:08.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:07 smithi006 bash[26022]: cluster 2023-11-17T05:38:06.546192+0000 mgr.smithi006.odexej (mgr.14176) 192 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:08.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:07 smithi156 bash[29655]: cluster 2023-11-17T05:38:06.546192+0000 mgr.smithi006.odexej (mgr.14176) 192 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:38:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:09 smithi006 bash[26022]: cluster 2023-11-17T05:38:08.546772+0000 mgr.smithi006.odexej (mgr.14176) 193 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:10.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:09 smithi156 bash[29655]: cluster 2023-11-17T05:38:08.546772+0000 mgr.smithi006.odexej (mgr.14176) 193 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:11.940 INFO:teuthology.orchestra.run.smithi006.stdout:128849018908 2023-11-17T05:38:11.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-17T05:38:12.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:11 smithi006 bash[26022]: cluster 2023-11-17T05:38:10.547370+0000 mgr.smithi006.odexej (mgr.14176) 194 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:12.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:11 smithi156 bash[29655]: cluster 2023-11-17T05:38:10.547370+0000 mgr.smithi006.odexej (mgr.14176) 194 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:14.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:13 smithi006 bash[26022]: cluster 2023-11-17T05:38:12.547987+0000 mgr.smithi006.odexej (mgr.14176) 195 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:14.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:13 smithi156 bash[29655]: cluster 2023-11-17T05:38:12.547987+0000 mgr.smithi006.odexej (mgr.14176) 195 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:16.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:15 smithi006 bash[26022]: cluster 2023-11-17T05:38:14.548580+0000 mgr.smithi006.odexej (mgr.14176) 196 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:16.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:15 smithi156 bash[29655]: cluster 2023-11-17T05:38:14.548580+0000 mgr.smithi006.odexej (mgr.14176) 196 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:16.795 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:38:17.663 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:17 smithi006 bash[26022]: cluster 2023-11-17T05:38:16.549180+0000 mgr.smithi006.odexej (mgr.14176) 197 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:17.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:17 smithi156 bash[29655]: cluster 2023-11-17T05:38:16.549180+0000 mgr.smithi006.odexej (mgr.14176) 197 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:19.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:19 smithi006 bash[26022]: cluster 2023-11-17T05:38:18.549778+0000 mgr.smithi006.odexej (mgr.14176) 198 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:19.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:19 smithi156 bash[29655]: cluster 2023-11-17T05:38:18.549778+0000 mgr.smithi006.odexej (mgr.14176) 198 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:20.099 INFO:teuthology.orchestra.run.smithi006.stdout:124554051613 2023-11-17T05:38:20.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-17T05:38:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:21 smithi006 bash[26022]: cluster 2023-11-17T05:38:20.550353+0000 mgr.smithi006.odexej (mgr.14176) 199 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:21.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:21 smithi156 bash[29655]: cluster 2023-11-17T05:38:20.550353+0000 mgr.smithi006.odexej (mgr.14176) 199 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:23.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:23 smithi006 bash[26022]: cluster 2023-11-17T05:38:22.550960+0000 mgr.smithi006.odexej (mgr.14176) 200 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:23.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:23 smithi156 bash[29655]: cluster 2023-11-17T05:38:22.550960+0000 mgr.smithi006.odexej (mgr.14176) 200 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:38:25.913 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:25 smithi006 bash[26022]: cluster 2023-11-17T05:38:24.551544+0000 mgr.smithi006.odexej (mgr.14176) 201 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:25.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:25 smithi156 bash[29655]: cluster 2023-11-17T05:38:24.551544+0000 mgr.smithi006.odexej (mgr.14176) 201 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:27.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:27 smithi006 bash[26022]: cluster 2023-11-17T05:38:26.552209+0000 mgr.smithi006.odexej (mgr.14176) 202 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:146028888093 2023-11-17T05:38:56.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-17T05:38:56.871 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:27 smithi156 bash[29655]: cluster 2023-11-17T05:38:26.552209+0000 mgr.smithi006.odexej (mgr.14176) 202 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:29 smithi156 bash[29655]: cluster 2023-11-17T05:38:28.552876+0000 mgr.smithi006.odexej (mgr.14176) 203 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:31 smithi156 bash[29655]: cluster 2023-11-17T05:38:30.553474+0000 mgr.smithi006.odexej (mgr.14176) 204 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:33 smithi156 bash[29655]: cluster 2023-11-17T05:38:32.554041+0000 mgr.smithi006.odexej (mgr.14176) 205 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:35 smithi156 bash[29655]: cluster 2023-11-17T05:38:34.554642+0000 mgr.smithi006.odexej (mgr.14176) 206 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.872 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:37 smithi156 bash[29655]: cluster 2023-11-17T05:38:36.555248+0000 mgr.smithi006.odexej (mgr.14176) 207 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:39 smithi156 bash[29655]: cluster 2023-11-17T05:38:38.555912+0000 mgr.smithi006.odexej (mgr.14176) 208 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:41 smithi156 bash[29655]: cluster 2023-11-17T05:38:40.556497+0000 mgr.smithi006.odexej (mgr.14176) 209 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:43 smithi156 bash[29655]: cluster 2023-11-17T05:38:42.557094+0000 mgr.smithi006.odexej (mgr.14176) 210 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:45 smithi156 bash[29655]: cluster 2023-11-17T05:38:44.557768+0000 mgr.smithi006.odexej (mgr.14176) 211 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:47 smithi156 bash[29655]: cluster 2023-11-17T05:38:46.558469+0000 mgr.smithi006.odexej (mgr.14176) 212 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.873 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:49 smithi156 bash[29655]: cluster 2023-11-17T05:38:48.559143+0000 mgr.smithi006.odexej (mgr.14176) 213 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.874 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:51 smithi156 bash[29655]: cluster 2023-11-17T05:38:50.559917+0000 mgr.smithi006.odexej (mgr.14176) 214 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.874 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:53 smithi156 bash[29655]: cluster 2023-11-17T05:38:52.560595+0000 mgr.smithi006.odexej (mgr.14176) 215 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.874 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:55 smithi156 bash[29655]: cluster 2023-11-17T05:38:54.561244+0000 mgr.smithi006.odexej (mgr.14176) 216 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.875 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:29 smithi006 bash[26022]: cluster 2023-11-17T05:38:28.552876+0000 mgr.smithi006.odexej (mgr.14176) 203 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.875 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:31 smithi006 bash[26022]: cluster 2023-11-17T05:38:30.553474+0000 mgr.smithi006.odexej (mgr.14176) 204 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:33 smithi006 bash[26022]: cluster 2023-11-17T05:38:32.554041+0000 mgr.smithi006.odexej (mgr.14176) 205 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:35 smithi006 bash[26022]: cluster 2023-11-17T05:38:34.554642+0000 mgr.smithi006.odexej (mgr.14176) 206 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:37 smithi006 bash[26022]: cluster 2023-11-17T05:38:36.555248+0000 mgr.smithi006.odexej (mgr.14176) 207 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:39 smithi006 bash[26022]: cluster 2023-11-17T05:38:38.555912+0000 mgr.smithi006.odexej (mgr.14176) 208 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:41 smithi006 bash[26022]: cluster 2023-11-17T05:38:40.556497+0000 mgr.smithi006.odexej (mgr.14176) 209 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:43 smithi006 bash[26022]: cluster 2023-11-17T05:38:42.557094+0000 mgr.smithi006.odexej (mgr.14176) 210 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:45 smithi006 bash[26022]: cluster 2023-11-17T05:38:44.557768+0000 mgr.smithi006.odexej (mgr.14176) 211 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:47 smithi006 bash[26022]: cluster 2023-11-17T05:38:46.558469+0000 mgr.smithi006.odexej (mgr.14176) 212 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:49 smithi006 bash[26022]: cluster 2023-11-17T05:38:48.559143+0000 mgr.smithi006.odexej (mgr.14176) 213 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:56.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:51 smithi006 bash[26022]: cluster 2023-11-17T05:38:50.559917+0000 mgr.smithi006.odexej (mgr.14176) 214 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:57.050 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:53 smithi006 bash[26022]: cluster 2023-11-17T05:38:52.560595+0000 mgr.smithi006.odexej (mgr.14176) 215 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:57.050 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:55 smithi006 bash[26022]: cluster 2023-11-17T05:38:54.561244+0000 mgr.smithi006.odexej (mgr.14176) 216 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:57.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:57 smithi006 bash[26022]: cluster 2023-11-17T05:38:56.561849+0000 mgr.smithi006.odexej (mgr.14176) 217 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:57.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:57 smithi156 bash[29655]: cluster 2023-11-17T05:38:56.561849+0000 mgr.smithi006.odexej (mgr.14176) 217 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:59.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:38:59 smithi006 bash[26022]: cluster 2023-11-17T05:38:58.562582+0000 mgr.smithi006.odexej (mgr.14176) 218 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:38:59.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:38:59 smithi156 bash[29655]: cluster 2023-11-17T05:38:58.562582+0000 mgr.smithi006.odexej (mgr.14176) 218 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:01.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:01 smithi006 bash[26022]: cluster 2023-11-17T05:39:00.563371+0000 mgr.smithi006.odexej (mgr.14176) 219 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:01.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:01 smithi156 bash[29655]: cluster 2023-11-17T05:39:00.563371+0000 mgr.smithi006.odexej (mgr.14176) 219 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:03.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:03 smithi006 bash[26022]: cluster 2023-11-17T05:39:02.564129+0000 mgr.smithi006.odexej (mgr.14176) 220 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:03.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:03 smithi156 bash[29655]: cluster 2023-11-17T05:39:02.564129+0000 mgr.smithi006.odexej (mgr.14176) 220 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:04.978 INFO:teuthology.orchestra.run.smithi006.stdout:68719476777 2023-11-17T05:39:05.556 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476758 got 68719476777 for osd.0 2023-11-17T05:39:05.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-17T05:39:05.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:05 smithi006 bash[26022]: cluster 2023-11-17T05:39:04.564789+0000 mgr.smithi006.odexej (mgr.14176) 221 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:05.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:05 smithi006 bash[26022]: audit 2023-11-17T05:39:04.976077+0000 mon.smithi006 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.6:0/3736817413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T05:39:05.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:05 smithi156 bash[29655]: cluster 2023-11-17T05:39:04.564789+0000 mgr.smithi006.odexej (mgr.14176) 221 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:05.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:05 smithi156 bash[29655]: audit 2023-11-17T05:39:04.976077+0000 mon.smithi006 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.6:0/3736817413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T05:39:07.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:07 smithi006 bash[26022]: cluster 2023-11-17T05:39:06.565438+0000 mgr.smithi006.odexej (mgr.14176) 222 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:07.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:07 smithi006 bash[26022]: audit 2023-11-17T05:39:06.717056+0000 mon.smithi006 (mon.0) 585 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:39:07.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:07 smithi156 bash[29655]: cluster 2023-11-17T05:39:06.565438+0000 mgr.smithi006.odexej (mgr.14176) 222 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:07.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:07 smithi156 bash[29655]: audit 2023-11-17T05:39:06.717056+0000 mon.smithi006 (mon.0) 585 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:39:09.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:09 smithi006 bash[26022]: cluster 2023-11-17T05:39:08.566170+0000 mgr.smithi006.odexej (mgr.14176) 223 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:09.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:09 smithi156 bash[29655]: cluster 2023-11-17T05:39:08.566170+0000 mgr.smithi006.odexej (mgr.14176) 223 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:11.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:11 smithi006 bash[26022]: cluster 2023-11-17T05:39:10.566783+0000 mgr.smithi006.odexej (mgr.14176) 224 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:11.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:11 smithi156 bash[29655]: cluster 2023-11-17T05:39:10.566783+0000 mgr.smithi006.odexej (mgr.14176) 224 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:13.207 INFO:teuthology.orchestra.run.smithi006.stdout:85899345962 2023-11-17T05:39:13.621 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:12.319667+0000 mon.smithi006 (mon.0) 586 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.621 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: cluster 2023-11-17T05:39:12.567256+0000 mgr.smithi006.odexej (mgr.14176) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:13.621 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:12.939117+0000 mon.smithi006 (mon.0) 587 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:13.203791+0000 mon.smithi006 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.6:0/1677888402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T05:39:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:13.236533+0000 mon.smithi006 (mon.0) 589 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:39:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:13.237841+0000 mon.smithi006 (mon.0) 590 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:39:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:13.247002+0000 mon.smithi006 (mon.0) 591 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:13 smithi156 bash[29655]: audit 2023-11-17T05:39:13.254270+0000 mon.smithi006 (mon.0) 592 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:39:13.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:12.319667+0000 mon.smithi006 (mon.0) 586 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: cluster 2023-11-17T05:39:12.567256+0000 mgr.smithi006.odexej (mgr.14176) 225 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:13.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:12.939117+0000 mon.smithi006 (mon.0) 587 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:13.203791+0000 mon.smithi006 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.6:0/1677888402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T05:39:13.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:13.236533+0000 mon.smithi006 (mon.0) 589 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:39:13.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:13.237841+0000 mon.smithi006 (mon.0) 590 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:39:13.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:13.247002+0000 mon.smithi006 (mon.0) 591 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:39:13.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:13 smithi006 bash[26022]: audit 2023-11-17T05:39:13.254270+0000 mon.smithi006 (mon.0) 592 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:39:13.815 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345944 got 85899345962 for osd.1 2023-11-17T05:39:13.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-17T05:39:15.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:15 smithi006 bash[26022]: cluster 2023-11-17T05:39:14.567868+0000 mgr.smithi006.odexej (mgr.14176) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:15.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:15 smithi156 bash[29655]: cluster 2023-11-17T05:39:14.567868+0000 mgr.smithi006.odexej (mgr.14176) 226 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:17.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:17 smithi006 bash[26022]: cluster 2023-11-17T05:39:16.568551+0000 mgr.smithi006.odexej (mgr.14176) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:17.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:17 smithi156 bash[29655]: cluster 2023-11-17T05:39:16.568551+0000 mgr.smithi006.odexej (mgr.14176) 227 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:18.674 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:19.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:19 smithi006 bash[26022]: cluster 2023-11-17T05:39:18.569247+0000 mgr.smithi006.odexej (mgr.14176) 228 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:19.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:19 smithi156 bash[29655]: cluster 2023-11-17T05:39:18.569247+0000 mgr.smithi006.odexej (mgr.14176) 228 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:21.503 INFO:teuthology.orchestra.run.smithi006.stdout:85899345964 2023-11-17T05:39:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:21 smithi006 bash[26022]: cluster 2023-11-17T05:39:20.569929+0000 mgr.smithi006.odexej (mgr.14176) 229 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:21.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:21 smithi006 bash[26022]: audit 2023-11-17T05:39:21.501217+0000 mon.smithi006 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.6:0/2841634748' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T05:39:21.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:21 smithi156 bash[29655]: cluster 2023-11-17T05:39:20.569929+0000 mgr.smithi006.odexej (mgr.14176) 229 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:21.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:21 smithi156 bash[29655]: audit 2023-11-17T05:39:21.501217+0000 mon.smithi006 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.6:0/2841634748' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T05:39:22.114 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345945 got 85899345964 for osd.2 2023-11-17T05:39:22.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-17T05:39:23.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:23 smithi006 bash[26022]: cluster 2023-11-17T05:39:22.570478+0000 mgr.smithi006.odexej (mgr.14176) 230 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:23.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:23 smithi156 bash[29655]: cluster 2023-11-17T05:39:22.570478+0000 mgr.smithi006.odexej (mgr.14176) 230 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:25.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:25 smithi006 bash[26022]: cluster 2023-11-17T05:39:24.571058+0000 mgr.smithi006.odexej (mgr.14176) 231 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:27.840 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:27.841 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:25 smithi156 bash[29655]: cluster 2023-11-17T05:39:24.571058+0000 mgr.smithi006.odexej (mgr.14176) 231 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:27.841 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:27 smithi156 bash[29655]: cluster 2023-11-17T05:39:26.571725+0000 mgr.smithi006.odexej (mgr.14176) 232 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:27.841 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:27 smithi006 bash[26022]: cluster 2023-11-17T05:39:26.571725+0000 mgr.smithi006.odexej (mgr.14176) 232 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:29.780 INFO:teuthology.orchestra.run.smithi006.stdout:107374182444 2023-11-17T05:39:29.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:29 smithi006 bash[26022]: cluster 2023-11-17T05:39:28.572309+0000 mgr.smithi006.odexej (mgr.14176) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:29.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:29 smithi156 bash[29655]: cluster 2023-11-17T05:39:28.572309+0000 mgr.smithi006.odexej (mgr.14176) 233 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:30.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182425 got 107374182444 for osd.3 2023-11-17T05:39:30.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-17T05:39:30.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:30 smithi006 bash[26022]: audit 2023-11-17T05:39:29.778232+0000 mon.smithi006 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.6:0/3423795327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T05:39:30.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:30 smithi156 bash[29655]: audit 2023-11-17T05:39:29.778232+0000 mon.smithi006 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.6:0/3423795327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T05:39:31.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:31 smithi006 bash[26022]: cluster 2023-11-17T05:39:30.573001+0000 mgr.smithi006.odexej (mgr.14176) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:31.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:31 smithi156 bash[29655]: cluster 2023-11-17T05:39:30.573001+0000 mgr.smithi006.odexej (mgr.14176) 234 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:33.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:33 smithi006 bash[26022]: cluster 2023-11-17T05:39:32.573711+0000 mgr.smithi006.odexej (mgr.14176) 235 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:33.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:33 smithi156 bash[29655]: cluster 2023-11-17T05:39:32.573711+0000 mgr.smithi006.odexej (mgr.14176) 235 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:35.246 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:35.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:35 smithi006 bash[26022]: cluster 2023-11-17T05:39:34.574409+0000 mgr.smithi006.odexej (mgr.14176) 236 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:35.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:35 smithi156 bash[29655]: cluster 2023-11-17T05:39:34.574409+0000 mgr.smithi006.odexej (mgr.14176) 236 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:37.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:37 smithi006 bash[26022]: cluster 2023-11-17T05:39:36.575070+0000 mgr.smithi006.odexej (mgr.14176) 237 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:37.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:37 smithi156 bash[29655]: cluster 2023-11-17T05:39:36.575070+0000 mgr.smithi006.odexej (mgr.14176) 237 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:38.040 INFO:teuthology.orchestra.run.smithi006.stdout:111669149740 2023-11-17T05:39:38.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:38 smithi006 bash[26022]: audit 2023-11-17T05:39:38.037759+0000 mon.smithi006 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.6:0/3558843583' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T05:39:38.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149721 got 111669149740 for osd.4 2023-11-17T05:39:38.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-17T05:39:38.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:38 smithi156 bash[29655]: audit 2023-11-17T05:39:38.037759+0000 mon.smithi006 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.6:0/3558843583' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T05:39:39.664 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:39 smithi006 bash[26022]: cluster 2023-11-17T05:39:38.575719+0000 mgr.smithi006.odexej (mgr.14176) 238 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:39.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:39 smithi156 bash[29655]: cluster 2023-11-17T05:39:38.575719+0000 mgr.smithi006.odexej (mgr.14176) 238 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:41.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:41 smithi006 bash[26022]: cluster 2023-11-17T05:39:40.576310+0000 mgr.smithi006.odexej (mgr.14176) 239 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:41.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:41 smithi156 bash[29655]: cluster 2023-11-17T05:39:40.576310+0000 mgr.smithi006.odexej (mgr.14176) 239 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:43.467 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:43.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:43 smithi006 bash[26022]: cluster 2023-11-17T05:39:42.576826+0000 mgr.smithi006.odexej (mgr.14176) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:43.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:43 smithi156 bash[29655]: cluster 2023-11-17T05:39:42.576826+0000 mgr.smithi006.odexej (mgr.14176) 240 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:45.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:45 smithi006 bash[26022]: cluster 2023-11-17T05:39:44.577479+0000 mgr.smithi006.odexej (mgr.14176) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:45.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:45 smithi156 bash[29655]: cluster 2023-11-17T05:39:44.577479+0000 mgr.smithi006.odexej (mgr.14176) 241 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:46.336 INFO:teuthology.orchestra.run.smithi006.stdout:128849018927 2023-11-17T05:39:46.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018927 for osd.5 2023-11-17T05:39:46.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-17T05:39:47.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:47 smithi006 bash[26022]: audit 2023-11-17T05:39:46.333252+0000 mon.smithi006 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.6:0/870075689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T05:39:47.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:47 smithi006 bash[26022]: cluster 2023-11-17T05:39:46.578225+0000 mgr.smithi006.odexej (mgr.14176) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:47.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:47 smithi156 bash[29655]: audit 2023-11-17T05:39:46.333252+0000 mon.smithi006 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.6:0/870075689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T05:39:47.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:47 smithi156 bash[29655]: cluster 2023-11-17T05:39:46.578225+0000 mgr.smithi006.odexej (mgr.14176) 242 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:49.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:49 smithi006 bash[26022]: cluster 2023-11-17T05:39:48.578986+0000 mgr.smithi006.odexej (mgr.14176) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:49.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:49 smithi156 bash[29655]: cluster 2023-11-17T05:39:48.578986+0000 mgr.smithi006.odexej (mgr.14176) 243 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:39:51.914 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:51 smithi006 bash[26022]: cluster 2023-11-17T05:39:50.579621+0000 mgr.smithi006.odexej (mgr.14176) 244 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:51.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:51 smithi156 bash[29655]: cluster 2023-11-17T05:39:50.579621+0000 mgr.smithi006.odexej (mgr.14176) 244 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:53.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:53 smithi006 bash[26022]: cluster 2023-11-17T05:39:52.580077+0000 mgr.smithi006.odexej (mgr.14176) 245 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:53.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:53 smithi156 bash[29655]: cluster 2023-11-17T05:39:52.580077+0000 mgr.smithi006.odexej (mgr.14176) 245 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:54.601 INFO:teuthology.orchestra.run.smithi006.stdout:124554051632 2023-11-17T05:39:54.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:54 smithi006 bash[26022]: audit 2023-11-17T05:39:54.598784+0000 mon.smithi006 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.6:0/3461233832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T05:39:54.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:54 smithi156 bash[29655]: audit 2023-11-17T05:39:54.598784+0000 mon.smithi006 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.6:0/3461233832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T05:39:55.192 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051613 got 124554051632 for osd.6 2023-11-17T05:39:55.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-17T05:39:55.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:55 smithi156 bash[29655]: cluster 2023-11-17T05:39:54.580602+0000 mgr.smithi006.odexej (mgr.14176) 246 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:56.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:55 smithi006 bash[26022]: cluster 2023-11-17T05:39:54.580602+0000 mgr.smithi006.odexej (mgr.14176) 246 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:57.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:57 smithi006 bash[26022]: cluster 2023-11-17T05:39:56.581167+0000 mgr.smithi006.odexej (mgr.14176) 247 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:57.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:57 smithi156 bash[29655]: cluster 2023-11-17T05:39:56.581167+0000 mgr.smithi006.odexej (mgr.14176) 247 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:59.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:39:59 smithi006 bash[26022]: cluster 2023-11-17T05:39:58.581735+0000 mgr.smithi006.odexej (mgr.14176) 248 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:39:59.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:39:59 smithi156 bash[29655]: cluster 2023-11-17T05:39:58.581735+0000 mgr.smithi006.odexej (mgr.14176) 248 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:00.047 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:40:00.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:00 smithi006 bash[26022]: cluster 2023-11-17T05:40:00.000166+0000 mon.smithi006 (mon.0) 598 : cluster [INF] overall HEALTH_OK 2023-11-17T05:40:00.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:00 smithi156 bash[29655]: cluster 2023-11-17T05:40:00.000166+0000 mon.smithi006 (mon.0) 598 : cluster [INF] overall HEALTH_OK 2023-11-17T05:40:01.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:01 smithi006 bash[26022]: cluster 2023-11-17T05:40:00.582244+0000 mgr.smithi006.odexej (mgr.14176) 249 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:01.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:01 smithi156 bash[29655]: cluster 2023-11-17T05:40:00.582244+0000 mgr.smithi006.odexej (mgr.14176) 249 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:02.830 INFO:teuthology.orchestra.run.smithi006.stdout:146028888112 2023-11-17T05:40:03.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888093 got 146028888112 for osd.7 2023-11-17T05:40:03.367 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-17T05:40:03.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T05:40:03.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:03 smithi006 bash[26022]: cluster 2023-11-17T05:40:02.582722+0000 mgr.smithi006.odexej (mgr.14176) 250 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:03.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:03 smithi006 bash[26022]: audit 2023-11-17T05:40:02.827353+0000 mon.smithi006 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.6:0/476460369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T05:40:03.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:03 smithi156 bash[29655]: cluster 2023-11-17T05:40:02.582722+0000 mgr.smithi006.odexej (mgr.14176) 250 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:03.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:03 smithi156 bash[29655]: audit 2023-11-17T05:40:02.827353+0000 mon.smithi006 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.6:0/476460369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T05:40:05.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:05 smithi006 bash[26022]: cluster 2023-11-17T05:40:04.583415+0000 mgr.smithi006.odexej (mgr.14176) 251 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:05.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:05 smithi156 bash[29655]: cluster 2023-11-17T05:40:04.583415+0000 mgr.smithi006.odexej (mgr.14176) 251 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:07.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:07 smithi006 bash[26022]: cluster 2023-11-17T05:40:06.583996+0000 mgr.smithi006.odexej (mgr.14176) 252 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:07.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:07 smithi156 bash[29655]: cluster 2023-11-17T05:40:06.583996+0000 mgr.smithi006.odexej (mgr.14176) 252 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:08.231 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:40:09.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:09 smithi006 bash[26022]: cluster 2023-11-17T05:40:08.584575+0000 mgr.smithi006.odexej (mgr.14176) 253 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:09.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:09 smithi156 bash[29655]: cluster 2023-11-17T05:40:08.584575+0000 mgr.smithi006.odexej (mgr.14176) 253 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:11.126 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:40:11.128 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-11-17T05:40:11.706 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2023-11-17T05:40:10.584894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749928448,"kb_used":2379512,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747548936,"statfs":{"total":767926730752,"available":765490110464,"internally_reserved":0,"allocated":2809856,"data_stored":1095316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-17T05:36:16.477001+0000","last_change":"2023-11-17T05:36:16.477001+0000","last_active":"2023-11-17T05:36:16.477001+0000","last_peered":"2023-11-17T05:36:16.477001+0000","last_clean":"2023-11-17T05:36:16.477001+0000","last_became_active":"2023-11-17T05:36:16.476555+0000","last_became_peered":"2023-11-17T05:36:16.476555+0000","last_unstale":"2023-11-17T05:36:16.477001+0000","last_undegraded":"2023-11-17T05:36:16.477001+0000","last_fullsized":"2023-11-17T05:36:16.477001+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-17T05:35:59.206337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-17T05:35:59.206337+0000","last_clean_scrub_stamp":"2023-11-17T05:35:59.206337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89200000000000002,"15min":0.89500000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.29,"5min":1.46,"15min":1.46},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.93500000000000005},"min":{"1min":0.71599999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.123,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":1.0349999999999999}]},{"osd":1,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.50700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.147,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.82999999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.095,"5min":1.357,"15min":1.357},"last":1.2789999999999999}]},{"osd":2,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.89700000000000002,"15min":0.90900000000000003},"min":{"1min":0.73399999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.052,"5min":1.242,"15min":1.242},"last":0.88100000000000001},{"interface":"front","average":{"1min":1,"5min":1.036,"15min":1.028},"min":{"1min":0.76100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4399999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":1.224}]},{"osd":3,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0029999999999999,"15min":0.97099999999999997},"min":{"1min":0.66200000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4370000000000001,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1,"15min":1.0149999999999999},"min":{"1min":0.63600000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.383,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.1599999999999999}]},{"osd":4,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0780000000000001,"15min":1.0860000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.1930000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.006},{"interface":"front","average":{"1min":1.173,"5min":1.1599999999999999,"15min":1.151},"min":{"1min":0.90700000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.524,"5min":2.379,"15min":2.379},"last":1.319}]},{"osd":5,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1679999999999999,"15min":1.1639999999999999},"min":{"1min":0.74399999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.5820000000000001,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.377},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.151,"15min":1.143},"min":{"1min":0.82099999999999995,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.482,"5min":1.744,"15min":1.744},"last":1.7110000000000001}]},{"osd":6,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2490000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":1.5489999999999999,"5min":1.581,"15min":1.581},"last":1.657},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.238,"15min":1.2490000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.389,"5min":1.804,"15min":1.804},"last":1.2549999999999999}]}]},{"osd":5,"up_from":30,"seq":128849018932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.022,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.753},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.69399999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.365,"5min":1.597,"15min":1.597},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80300000000000005,"15min":0.78300000000000003},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.85799999999999998,"15min":0.84999999999999998},"min":{"1min":0.70799999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2769999999999999,"5min":1.5,"15min":1.5},"last":0.93500000000000005}]},{"osd":2,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.008,"15min":1.0069999999999999},"min":{"1min":0.81100000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.119,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.077},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95799999999999996,"15min":0.94499999999999995},"min":{"1min":0.76600000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.25,"5min":1.51,"15min":1.51},"last":0.88}]},{"osd":3,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.65000000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0649999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.55900000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.375,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.278}]},{"osd":4,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0229999999999999,"15min":1.0249999999999999},"min":{"1min":0.77100000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.264,"5min":1.405,"15min":1.405},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.139},"min":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.431,"5min":1.984,"15min":1.984},"last":1.1839999999999999}]},{"osd":6,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1499999999999999,"15min":1.1419999999999999},"min":{"1min":0.94999999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.246,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.1890000000000001,"15min":1.1899999999999999},"min":{"1min":0.88100000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3680000000000001,"5min":1.552,"15min":1.552},"last":1.2150000000000001}]},{"osd":7,"last update":"Fri Nov 17 05:39:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1259999999999999,"15min":1.1160000000000001},"min":{"1min":0.80500000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.4510000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.1899999999999999,"15min":1.1839999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3819999999999999,"5min":1.778,"15min":1.778},"last":1.2450000000000001}]}]},{"osd":1,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297436,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443620,"statfs":{"total":95990841344,"available":95686266880,"internally_reserved":0,"allocated":348160,"data_stored":136725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.81699999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.145,"5min":1.274,"15min":1.274},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.61399999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.177,"5min":1.262,"15min":1.262},"last":0.97399999999999998}]},{"osd":2,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86299999999999999,"15min":0.88900000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.087,"5min":1.381,"15min":1.381},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.95899999999999996,"15min":0.94899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3180000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.79100000000000004}]},{"osd":3,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2150000000000001,"5min":1.605,"15min":1.605},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.876,"15min":0.89100000000000001},"min":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.177,"5min":1.464,"15min":1.464},"last":1.177}]},{"osd":4,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.075,"15min":1.0680000000000001},"min":{"1min":0.81000000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.84099999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3260000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.095}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3240000000000001,"5min":1.522,"15min":1.522},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.131,"15min":1.1200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4430000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.21}]},{"osd":6,"last update":"Fri Nov 17 05:39:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0700000000000001,"15min":1.083},"min":{"1min":0.69299999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.3939999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.27,"15min":1.26},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5149999999999999,"5min":1.732,"15min":1.732},"last":1.1399999999999999}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.288,"15min":1.286},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.45,"5min":1.595,"15min":1.595},"last":1.236},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2410000000000001,"15min":1.24},"min":{"1min":0.79400000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.5289999999999999,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.27}]}]},{"osd":0,"up_from":16,"seq":68719476790,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.83399999999999996,"15min":0.80000000000000004},"min":{"1min":0.55500000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0189999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.022},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.58599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0169999999999999,"5min":1.105,"15min":1.105},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.72199999999999998},"min":{"1min":0.51200000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.141,"5min":1.165,"15min":1.165},"last":0.746}]},{"osd":3,"last update":"Fri Nov 17 05:39:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.871,"15min":0.85699999999999998},"min":{"1min":0.64900000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.133,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0529999999999999,"15min":1.0429999999999999},"min":{"1min":0.877,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2609999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.1339999999999999}]},{"osd":4,"last update":"Fri Nov 17 05:39:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.88500000000000001,"15min":0.89000000000000001},"min":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.028,"5min":1.016,"15min":1},"min":{"1min":0.78900000000000003,"5min":0.629,"15min":0.629},"max":{"1min":1.1850000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.97099999999999997}]},{"osd":5,"last update":"Fri Nov 17 05:39:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98999999999999999,"15min":0.98299999999999998},"min":{"1min":0.73599999999999999,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.1930000000000001,"5min":1.357,"15min":1.357},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.095,"15min":1.0800000000000001},"min":{"1min":0.80500000000000005,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.248}]},{"osd":6,"last update":"Fri Nov 17 05:39:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.024,"15min":1.0109999999999999},"min":{"1min":0.76800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2749999999999999,"5min":1.321,"15min":1.321},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.161,"5min":1.1339999999999999,"15min":1.1120000000000001},"min":{"1min":0.90800000000000003,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.343,"5min":1.431,"15min":1.431},"last":1.321}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1719999999999999,"15min":1.155},"min":{"1min":1.03,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.363,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.2609999999999999,"15min":1.244},"min":{"1min":1.0609999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":1.369}]}]},{"osd":2,"up_from":20,"seq":85899345974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.53400000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0960000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.60699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.68100000000000005}]},{"osd":1,"last update":"Fri Nov 17 05:40:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.85899999999999999,"15min":0.84099999999999997},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1930000000000001,"5min":1.343,"15min":1.343},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.89400000000000002,"15min":0.89000000000000001},"min":{"1min":0.70799999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0600000000000001,"5min":1.625,"15min":1.625},"last":0.745}]},{"osd":3,"last update":"Fri Nov 17 05:39:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96399999999999997,"15min":0.93799999999999994},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1859999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.0189999999999999,"15min":1.04},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.1080000000000001,"5min":1.53,"15min":1.53},"last":0.95799999999999996}]},{"osd":4,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.877},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.044,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.026,"5min":1.0649999999999999,"15min":1.0800000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.29,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.1259999999999999,"5min":1.222,"15min":1.222},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.109,"15min":1.123},"min":{"1min":0.86099999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.4279999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.1439999999999999}]},{"osd":6,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.014,"15min":1.042},"min":{"1min":0.77200000000000002,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.0760000000000001,"5min":1.458,"15min":1.458},"last":1.002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.093,"15min":1.119},"min":{"1min":0.83899999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.4259999999999999,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.88400000000000001}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1419999999999999,"15min":1.1519999999999999},"min":{"1min":1.0269999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.228,"15min":1.236},"min":{"1min":1.0149999999999999,"5min":0.874,"15min":0.874},"max":{"1min":1.5489999999999999,"5min":1.659,"15min":1.659},"last":1.5149999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297436,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443620,"statfs":{"total":95990841344,"available":95686266880,"internally_reserved":0,"allocated":348160,"data_stored":136351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.874,"15min":0.91400000000000003},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.044,"5min":3.097,"15min":3.097},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.92600000000000005,"15min":0.90800000000000003},"min":{"1min":0.70699999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.79800000000000004}]},{"osd":1,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.876,"15min":0.90800000000000003},"min":{"1min":0.59099999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.1100000000000001,"5min":1.423,"15min":1.423},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.75,"15min":0.72399999999999998},"min":{"1min":0.54600000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0920000000000001,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.746}]},{"osd":2,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.67100000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.173,"5min":1.246,"15min":1.246},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":1.0209999999999999},"min":{"1min":0.67500000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.419,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.97699999999999998}]},{"osd":4,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0409999999999999,"15min":0.98699999999999999},"min":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4670000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.85299999999999998}]},{"osd":5,"last update":"Fri Nov 17 05:39:13 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.091,"15min":1.1000000000000001},"min":{"1min":0.66400000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3320000000000001,"5min":3.343,"15min":3.343},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.871,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.399,"5min":1.974,"15min":1.974},"last":0.93500000000000005}]},{"osd":6,"last update":"Fri Nov 17 05:39:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.159,"15min":1.131},"min":{"1min":1.1120000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4370000000000001,"5min":1.468,"15min":1.468},"last":1.252},{"interface":"front","average":{"1min":1.129,"5min":1.05,"15min":1.03},"min":{"1min":0.94799999999999995,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":1.292}]},{"osd":7,"last update":"Fri Nov 17 05:39:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2709999999999999},"min":{"1min":0.96599999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.46,"5min":3.407,"15min":3.407},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":1.0449999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.4690000000000001,"5min":2.698,"15min":2.698},"last":1.3620000000000001}]}]},{"osd":4,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.77800000000000002,"15min":0.82399999999999995},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.0049999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.871,"15min":0.92100000000000004},"min":{"1min":0.68500000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1870000000000001,"5min":1.206,"15min":1.206},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.85999999999999999,"15min":0.87},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95499999999999996,"15min":0.95899999999999996},"min":{"1min":0.66600000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.101,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0489999999999999}]},{"osd":2,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.84399999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.94599999999999995,"5min":1.222,"15min":1.222},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.63,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.77900000000000003}]},{"osd":3,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.996,"15min":0.998},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.75800000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.1279999999999999,"5min":1.331,"15min":1.331},"last":0.90000000000000002}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.008,"15min":1.0049999999999999},"min":{"1min":0.78700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.2689999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0860000000000001,"15min":1.0820000000000001},"min":{"1min":0.82799999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":1.151}]},{"osd":6,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0629999999999999,"15min":1.0580000000000001},"min":{"1min":0.73699999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.107},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.085,"15min":1.109},"min":{"1min":0.80000000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.3200000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Nov 17 05:39:21 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1539999999999999,"15min":1.1579999999999999},"min":{"1min":1.0249999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.167,"15min":1.1559999999999999},"min":{"1min":1.0509999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.2849999999999999}]}]},{"osd":6,"up_from":29,"seq":124554051635,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:39:10 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.56899999999999995,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0660000000000001,"5min":1.244,"15min":1.244},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.84299999999999997,"15min":0.873},"min":{"1min":0.57299999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0269999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.82399999999999995}]},{"osd":1,"last update":"Fri Nov 17 05:39:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.872,"15min":0.88100000000000001},"min":{"1min":0.70299999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.155,"5min":1.175,"15min":1.175},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.96599999999999997,"15min":0.96199999999999997},"min":{"1min":0.78500000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.165,"5min":1.331,"15min":1.331},"last":0.78800000000000003}]},{"osd":2,"last update":"Fri Nov 17 05:39:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.89700000000000002,"15min":0.88500000000000001},"min":{"1min":0.67700000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.298,"5min":1.399,"15min":1.399},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.879,"15min":0.88600000000000001},"min":{"1min":0.66100000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0129999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.92500000000000004}]},{"osd":3,"last update":"Fri Nov 17 05:39:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90100000000000002,"15min":0.88500000000000001},"min":{"1min":0.79300000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.2030000000000001,"5min":1.546,"15min":1.546},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.0600000000000001,"15min":1.0449999999999999},"min":{"1min":0.82699999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.339,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.0760000000000001}]},{"osd":4,"last update":"Fri Nov 17 05:39:10 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.99199999999999999,"15min":0.998},"min":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.2410000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.034,"15min":1.0269999999999999},"min":{"1min":0.79800000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.27,"5min":1.319,"15min":1.319},"last":1.0469999999999999}]},{"osd":5,"last update":"Fri Nov 17 05:39:14 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.1990000000000001,"15min":1.1910000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.137},{"interface":"front","average":{"1min":1.258,"5min":1.1719999999999999,"15min":1.1599999999999999},"min":{"1min":0.94099999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":0.96599999999999997}]},{"osd":7,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1859999999999999,"15min":1.1850000000000001},"min":{"1min":1.024,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.367,"5min":1.51,"15min":1.51},"last":1.425},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.1919999999999999,"15min":1.169},"min":{"1min":1.0589999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.117}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T05:40:11.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T05:40:11.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:11 smithi006 bash[26022]: cluster 2023-11-17T05:40:10.585141+0000 mgr.smithi006.odexej (mgr.14176) 254 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:11.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:11 smithi156 bash[29655]: cluster 2023-11-17T05:40:10.585141+0000 mgr.smithi006.odexej (mgr.14176) 254 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:12.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:12 smithi006 bash[26022]: audit 2023-11-17T05:40:11.115336+0000 mgr.smithi006.odexej (mgr.14176) 255 : audit [DBG] from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:40:12.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:12 smithi156 bash[29655]: audit 2023-11-17T05:40:11.115336+0000 mgr.smithi006.odexej (mgr.14176) 255 : audit [DBG] from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:40:13.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:13 smithi006 bash[26022]: cluster 2023-11-17T05:40:12.585629+0000 mgr.smithi006.odexej (mgr.14176) 256 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:13.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:13 smithi006 bash[26022]: audit 2023-11-17T05:40:13.266748+0000 mon.smithi006 (mon.0) 600 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:40:13.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:13 smithi156 bash[29655]: cluster 2023-11-17T05:40:12.585629+0000 mgr.smithi006.odexej (mgr.14176) 256 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:13.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:13 smithi156 bash[29655]: audit 2023-11-17T05:40:13.266748+0000 mon.smithi006 (mon.0) 600 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:40:15.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:15 smithi156 bash[29655]: cluster 2023-11-17T05:40:14.586207+0000 mgr.smithi006.odexej (mgr.14176) 257 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:16.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:15 smithi006 bash[26022]: cluster 2023-11-17T05:40:14.586207+0000 mgr.smithi006.odexej (mgr.14176) 257 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:16.567 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:40:17.652 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:17 smithi006 bash[26022]: cluster 2023-11-17T05:40:16.586824+0000 mgr.smithi006.odexej (mgr.14176) 258 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:17.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:17 smithi156 bash[29655]: cluster 2023-11-17T05:40:16.586824+0000 mgr.smithi006.odexej (mgr.14176) 258 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:19.310 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:40:19.311 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-11-17T05:40:19.845 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2023-11-17T05:40:18.587102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749928448,"kb_used":2379512,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747548936,"statfs":{"total":767926730752,"available":765490110464,"internally_reserved":0,"allocated":2809856,"data_stored":1095316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-17T05:36:16.477001+0000","last_change":"2023-11-17T05:36:16.477001+0000","last_active":"2023-11-17T05:36:16.477001+0000","last_peered":"2023-11-17T05:36:16.477001+0000","last_clean":"2023-11-17T05:36:16.477001+0000","last_became_active":"2023-11-17T05:36:16.476555+0000","last_became_peered":"2023-11-17T05:36:16.476555+0000","last_unstale":"2023-11-17T05:36:16.477001+0000","last_undegraded":"2023-11-17T05:36:16.477001+0000","last_fullsized":"2023-11-17T05:36:16.477001+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-17T05:35:59.206337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-17T05:35:59.206337+0000","last_clean_scrub_stamp":"2023-11-17T05:35:59.206337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":34,"seq":146028888115,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.89200000000000002,"15min":0.89500000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.29,"5min":1.46,"15min":1.46},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.93500000000000005},"min":{"1min":0.71599999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.123,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.79000000000000004}]},{"osd":1,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.50700000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.147,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.82999999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.095,"5min":1.357,"15min":1.357},"last":1.1970000000000001}]},{"osd":2,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.89700000000000002,"15min":0.90900000000000003},"min":{"1min":0.73399999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.052,"5min":1.242,"15min":1.242},"last":0.81999999999999995},{"interface":"front","average":{"1min":1,"5min":1.036,"15min":1.028},"min":{"1min":0.76100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4399999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":0.94999999999999996}]},{"osd":3,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0029999999999999,"15min":0.97099999999999997},"min":{"1min":0.66200000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4370000000000001,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":0.878},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1,"15min":1.0149999999999999},"min":{"1min":0.63600000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.383,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.90800000000000003}]},{"osd":4,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0780000000000001,"15min":1.0860000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.1930000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.1599999999999999,"15min":1.151},"min":{"1min":0.90700000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.524,"5min":2.379,"15min":2.379},"last":1.089}]},{"osd":5,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1679999999999999,"15min":1.1639999999999999},"min":{"1min":0.74399999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.5820000000000001,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.151,"15min":1.143},"min":{"1min":0.82099999999999995,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.482,"5min":1.744,"15min":1.744},"last":1.2629999999999999}]},{"osd":6,"last update":"Fri Nov 17 05:39:19 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2490000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":1.5489999999999999,"5min":1.581,"15min":1.581},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.238,"15min":1.2490000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.389,"5min":1.804,"15min":1.804},"last":1.0720000000000001}]}]},{"osd":5,"up_from":30,"seq":128849018933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.022,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.69399999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.365,"5min":1.597,"15min":1.597},"last":0.82999999999999996}]},{"osd":1,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80300000000000005,"15min":0.78300000000000003},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.441},"max":{"1min":1.228,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.85799999999999998,"15min":0.84999999999999998},"min":{"1min":0.70799999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2769999999999999,"5min":1.5,"15min":1.5},"last":0.89400000000000002}]},{"osd":2,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.008,"15min":1.0069999999999999},"min":{"1min":0.81100000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.119,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.95799999999999996,"15min":0.94499999999999995},"min":{"1min":0.76600000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.25,"5min":1.51,"15min":1.51},"last":1.012}]},{"osd":3,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.65000000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0649999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.95599999999999996,"15min":0.96599999999999997},"min":{"1min":0.55900000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.375,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0229999999999999,"15min":1.0249999999999999},"min":{"1min":0.77100000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.264,"5min":1.405,"15min":1.405},"last":1.117},{"interface":"front","average":{"1min":1.131,"5min":1.133,"15min":1.139},"min":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.431,"5min":1.984,"15min":1.984},"last":1.036}]},{"osd":6,"last update":"Fri Nov 17 05:39:15 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1499999999999999,"15min":1.1419999999999999},"min":{"1min":0.94999999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.246,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.1890000000000001,"15min":1.1899999999999999},"min":{"1min":0.88100000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3680000000000001,"5min":1.552,"15min":1.552},"last":1.3859999999999999}]},{"osd":7,"last update":"Fri Nov 17 05:39:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1259999999999999,"15min":1.1160000000000001},"min":{"1min":0.80500000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.4510000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.319},{"interface":"front","average":{"1min":1.169,"5min":1.1899999999999999,"15min":1.1839999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3819999999999999,"5min":1.778,"15min":1.778},"last":1.1799999999999999}]}]},{"osd":1,"up_from":20,"seq":85899345975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297436,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443620,"statfs":{"total":95990841344,"available":95686266880,"internally_reserved":0,"allocated":348160,"data_stored":136725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.81699999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.145,"5min":1.274,"15min":1.274},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.61399999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.177,"5min":1.262,"15min":1.262},"last":1.0980000000000001}]},{"osd":2,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86299999999999999,"15min":0.88900000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.087,"5min":1.381,"15min":1.381},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.95899999999999996,"15min":0.94899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3180000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.2749999999999999}]},{"osd":3,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2150000000000001,"5min":1.605,"15min":1.605},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.876,"15min":0.89100000000000001},"min":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.177,"5min":1.464,"15min":1.464},"last":0.77000000000000002}]},{"osd":4,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.075,"15min":1.0680000000000001},"min":{"1min":0.81000000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.84099999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3260000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.99399999999999999}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.103,"15min":1.1000000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3240000000000001,"5min":1.522,"15min":1.522},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.131,"15min":1.1200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4430000000000001,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.21}]},{"osd":6,"last update":"Fri Nov 17 05:40:12 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.069,"15min":1.083},"min":{"1min":0.80900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5329999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.02},{"interface":"front","average":{"1min":1.228,"5min":1.2649999999999999,"15min":1.258},"min":{"1min":1.014,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5880000000000001,"5min":1.732,"15min":1.732},"last":1.169}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.288,"15min":1.286},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.45,"5min":1.595,"15min":1.595},"last":1.387},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2410000000000001,"15min":1.24},"min":{"1min":0.79400000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.5289999999999999,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.323}]}]},{"osd":0,"up_from":16,"seq":68719476791,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.83399999999999996,"15min":0.80000000000000004},"min":{"1min":0.55500000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0189999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.58599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0169999999999999,"5min":1.105,"15min":1.105},"last":1.133}]},{"osd":2,"last update":"Fri Nov 17 05:40:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.74399999999999999,"15min":0.72199999999999998},"min":{"1min":0.51200000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.141,"5min":1.165,"15min":1.165},"last":0.81200000000000006}]},{"osd":3,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88900000000000001,"15min":0.86299999999999999},"min":{"1min":0.72199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2310000000000001,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0469999999999999,"15min":1.0409999999999999},"min":{"1min":0.79700000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.139,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.101}]},{"osd":4,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.871,"15min":0.88500000000000001},"min":{"1min":0.624,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.147,"5min":1.264,"15min":1.264},"last":1.044},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0189999999999999,"15min":1.0009999999999999},"min":{"1min":0.86699999999999999,"5min":0.629,"15min":0.629},"max":{"1min":1.304,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.995}]},{"osd":5,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.996,"15min":0.98499999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.288,"5min":1.357,"15min":1.357},"last":1.288},{"interface":"front","average":{"1min":1.083,"5min":1.097,"15min":1.081},"min":{"1min":0.92400000000000004,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.2969999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.179}]},{"osd":6,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.79400000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.254,"5min":1.321,"15min":1.321},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1479999999999999,"15min":1.1160000000000001},"min":{"1min":1.042,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.321,"5min":1.431,"15min":1.431},"last":1.2430000000000001}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1719999999999999,"15min":1.155},"min":{"1min":1.03,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.363,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.379},{"interface":"front","average":{"1min":1.278,"5min":1.2609999999999999,"15min":1.244},"min":{"1min":1.0609999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":1.327}]}]},{"osd":2,"up_from":20,"seq":85899345975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.53400000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0960000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.60699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Nov 17 05:40:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.85899999999999999,"15min":0.84099999999999997},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1930000000000001,"5min":1.343,"15min":1.343},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.89400000000000002,"15min":0.89000000000000001},"min":{"1min":0.70799999999999996,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0600000000000001,"5min":1.625,"15min":1.625},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Nov 17 05:40:11 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.98699999999999999,"15min":0.94499999999999995},"min":{"1min":0.85699999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.089},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.004,"15min":1.0349999999999999},"min":{"1min":0.81200000000000006,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.1220000000000001,"5min":1.53,"15min":1.53},"last":1.018}]},{"osd":4,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.877},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.044,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.0649999999999999,"15min":1.0800000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.29,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.89300000000000002}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.1259999999999999,"5min":1.222,"15min":1.222},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.109,"15min":1.123},"min":{"1min":0.86099999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.4279999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.0369999999999999}]},{"osd":6,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.014,"15min":1.042},"min":{"1min":0.77200000000000002,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.0760000000000001,"5min":1.458,"15min":1.458},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.093,"15min":1.119},"min":{"1min":0.83899999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.4259999999999999,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Fri Nov 17 05:39:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1419999999999999,"15min":1.1519999999999999},"min":{"1min":1.0269999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.228,"15min":1.236},"min":{"1min":1.0149999999999999,"5min":0.874,"15min":0.874},"max":{"1min":1.5489999999999999,"5min":1.659,"15min":1.659},"last":1.133}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297436,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443620,"statfs":{"total":95990841344,"available":95686266880,"internally_reserved":0,"allocated":348160,"data_stored":136351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.874,"15min":0.91400000000000003},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.044,"5min":3.097,"15min":3.097},"last":1.238},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.92600000000000005,"15min":0.90800000000000003},"min":{"1min":0.70699999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.876,"15min":0.90800000000000003},"min":{"1min":0.59099999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.1100000000000001,"5min":1.423,"15min":1.423},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.75,"15min":0.72399999999999998},"min":{"1min":0.54600000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.0920000000000001,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.77200000000000002}]},{"osd":2,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.89900000000000002,"15min":0.875},"min":{"1min":0.67100000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.173,"5min":1.246,"15min":1.246},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.99099999999999999,"15min":1.0209999999999999},"min":{"1min":0.67500000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.419,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.335}]},{"osd":4,"last update":"Fri Nov 17 05:40:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0409999999999999,"15min":0.98699999999999999},"min":{"1min":0.80500000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4670000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.071,"15min":1.093},"min":{"1min":0.63200000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5009999999999999,"5min":3.343,"15min":3.343},"last":1.008},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.087,"15min":1.079},"min":{"1min":0.61699999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.444,"5min":1.974,"15min":1.974},"last":1.0920000000000001}]},{"osd":6,"last update":"Fri Nov 17 05:40:14 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.181,"15min":1.139},"min":{"1min":0.93700000000000006,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.161,"5min":1.0780000000000001,"15min":1.04},"min":{"1min":0.78700000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.048}]},{"osd":7,"last update":"Fri Nov 17 05:39:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2709999999999999},"min":{"1min":0.96599999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.46,"5min":3.407,"15min":3.407},"last":1.133},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":1.0449999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.4690000000000001,"5min":2.698,"15min":2.698},"last":1.3939999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.77800000000000002,"15min":0.82399999999999995},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.0049999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.871,"15min":0.92100000000000004},"min":{"1min":0.68500000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1870000000000001,"5min":1.206,"15min":1.206},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.85999999999999999,"15min":0.87},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95499999999999996,"15min":0.95899999999999996},"min":{"1min":0.66600000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.101,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0489999999999999}]},{"osd":2,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.84399999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.94599999999999995,"5min":1.222,"15min":1.222},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.63,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.77900000000000003}]},{"osd":3,"last update":"Fri Nov 17 05:40:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.996,"15min":0.998},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.75800000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.1279999999999999,"5min":1.331,"15min":1.331},"last":0.90000000000000002}]},{"osd":5,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.008,"15min":1.0049999999999999},"min":{"1min":0.78700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.2689999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0860000000000001,"15min":1.0820000000000001},"min":{"1min":0.82799999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":1.151}]},{"osd":6,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0629999999999999,"15min":1.0580000000000001},"min":{"1min":0.73699999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.107},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.085,"15min":1.109},"min":{"1min":0.80000000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.3200000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Nov 17 05:39:21 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1539999999999999,"15min":1.1579999999999999},"min":{"1min":1.0249999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.167,"15min":1.1559999999999999},"min":{"1min":1.0509999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.2849999999999999}]}]},{"osd":6,"up_from":29,"seq":124554051637,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297440,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443616,"statfs":{"total":95990841344,"available":95686262784,"internally_reserved":0,"allocated":352256,"data_stored":137040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 05:40:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.49099999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0960000000000001,"5min":1.244,"15min":1.244},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.86899999999999999},"min":{"1min":0.59999999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.177,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Fri Nov 17 05:40:13 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.67100000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.1000000000000001,"5min":1.175,"15min":1.175},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.96399999999999997,"15min":0.96199999999999997},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.2769999999999999,"5min":1.331,"15min":1.331},"last":1.099}]},{"osd":2,"last update":"Fri Nov 17 05:40:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.88900000000000001},"min":{"1min":0.623,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3049999999999999,"5min":1.399,"15min":1.399},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.88400000000000001},"min":{"1min":0.59599999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Nov 17 05:40:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.91200000000000003,"15min":0.88900000000000001},"min":{"1min":0.749,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.2470000000000001,"5min":1.546,"15min":1.546},"last":0.75},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0640000000000001,"15min":1.046},"min":{"1min":0.72999999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.93700000000000006}]},{"osd":4,"last update":"Fri Nov 17 05:40:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.97999999999999998,"15min":0.99399999999999999},"min":{"1min":0.68200000000000005,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1859999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.01,"5min":1.0309999999999999,"15min":1.026},"min":{"1min":0.73399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.871}]},{"osd":5,"last update":"Fri Nov 17 05:40:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.196,"15min":1.1899999999999999},"min":{"1min":0.82899999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.4750000000000001,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.181},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.175,"15min":1.161},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.7150000000000001,"5min":1.913,"15min":1.913},"last":1.052}]},{"osd":7,"last update":"Fri Nov 17 05:39:16 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1859999999999999,"15min":1.1850000000000001},"min":{"1min":1.024,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.367,"5min":1.51,"15min":1.51},"last":1.079},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.1919999999999999,"15min":1.169},"min":{"1min":1.0589999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.1339999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T05:40:19.846 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-17T05:40:19.846 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-17T05:40:19.846 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-17T05:40:19.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph health --format=json 2023-11-17T05:40:20.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: cluster 2023-11-17T05:40:18.587373+0000 mgr.smithi006.odexej (mgr.14176) 259 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:20.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:18.894100+0000 mon.smithi006 (mon.0) 601 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:18.993271+0000 mon.smithi006 (mon.0) 602 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:19.272729+0000 mon.smithi006 (mon.0) 603 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:40:20.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:19.274355+0000 mon.smithi006 (mon.0) 604 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:40:20.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:19.282227+0000 mon.smithi006 (mon.0) 605 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:19 smithi006 bash[26022]: audit 2023-11-17T05:40:19.293574+0000 mon.smithi006 (mon.0) 606 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:40:20.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: cluster 2023-11-17T05:40:18.587373+0000 mgr.smithi006.odexej (mgr.14176) 259 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:20.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:18.894100+0000 mon.smithi006 (mon.0) 601 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:18.993271+0000 mon.smithi006 (mon.0) 602 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:19.272729+0000 mon.smithi006 (mon.0) 603 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:40:20.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:19.274355+0000 mon.smithi006 (mon.0) 604 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:40:20.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:19.282227+0000 mon.smithi006 (mon.0) 605 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:20.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:19 smithi156 bash[29655]: audit 2023-11-17T05:40:19.293574+0000 mon.smithi006 (mon.0) 606 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:40:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:20 smithi006 bash[26022]: audit 2023-11-17T05:40:19.302809+0000 mgr.smithi006.odexej (mgr.14176) 260 : audit [DBG] from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:40:21.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:20 smithi156 bash[29655]: audit 2023-11-17T05:40:19.302809+0000 mgr.smithi006.odexej (mgr.14176) 260 : audit [DBG] from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:40:22.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:21 smithi006 bash[26022]: cluster 2023-11-17T05:40:20.588008+0000 mgr.smithi006.odexej (mgr.14176) 261 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:21 smithi156 bash[29655]: cluster 2023-11-17T05:40:20.588008+0000 mgr.smithi006.odexej (mgr.14176) 261 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:24.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:23 smithi006 bash[26022]: cluster 2023-11-17T05:40:22.588585+0000 mgr.smithi006.odexej (mgr.14176) 262 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:24.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:23 smithi156 bash[29655]: cluster 2023-11-17T05:40:22.588585+0000 mgr.smithi006.odexej (mgr.14176) 262 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/mon.smithi006/config 2023-11-17T05:40:26.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:25 smithi006 bash[26022]: cluster 2023-11-17T05:40:24.589201+0000 mgr.smithi006.odexej (mgr.14176) 263 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:26.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:25 smithi156 bash[29655]: cluster 2023-11-17T05:40:24.589201+0000 mgr.smithi006.odexej (mgr.14176) 263 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:27 smithi006 bash[26022]: cluster 2023-11-17T05:40:26.589786+0000 mgr.smithi006.odexej (mgr.14176) 264 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:27.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:40:27.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:27 smithi156 bash[29655]: cluster 2023-11-17T05:40:26.589786+0000 mgr.smithi006.odexej (mgr.14176) 264 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:28.141 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-17T05:40:28.141 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-17T05:40:28.141 INFO:tasks.cephadm:Setup complete, yielding 2023-11-17T05:40:28.142 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T05:40:28.151 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T05:40:28.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-17T05:40:28.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:28 smithi006 bash[26022]: audit 2023-11-17T05:40:27.554462+0000 mon.smithi006 (mon.0) 607 : audit [DBG] from='client.? 172.21.15.6:0/4242003645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T05:40:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:28 smithi156 bash[29655]: audit 2023-11-17T05:40:27.554462+0000 mon.smithi006 (mon.0) 607 : audit [DBG] from='client.? 172.21.15.6:0/4242003645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T05:40:29.428 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:29 smithi006 bash[26022]: cluster 2023-11-17T05:40:28.590471+0000 mgr.smithi006.odexej (mgr.14176) 265 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:29.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:29 smithi156 bash[29655]: cluster 2023-11-17T05:40:28.590471+0000 mgr.smithi006.odexej (mgr.14176) 265 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:31.136 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: cephadm 2023-11-17T05:40:31.136 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2023-11-17T05:40:31.136 INFO:teuthology.orchestra.run.smithi006.stdout:Paused: No 2023-11-17T05:40:31.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-17T05:40:31.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:31 smithi006 bash[26022]: cluster 2023-11-17T05:40:30.591107+0000 mgr.smithi006.odexej (mgr.14176) 266 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:31.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:31 smithi156 bash[29655]: cluster 2023-11-17T05:40:30.591107+0000 mgr.smithi006.odexej (mgr.14176) 266 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:32.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:32 smithi006 bash[26022]: audit 2023-11-17T05:40:31.132179+0000 mgr.smithi006.odexej (mgr.14176) 267 : audit [DBG] from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:32.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:32 smithi156 bash[29655]: audit 2023-11-17T05:40:31.132179+0000 mgr.smithi006.odexej (mgr.14176) 267 : audit [DBG] from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:33.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:33 smithi006 bash[26022]: cluster 2023-11-17T05:40:32.591526+0000 mgr.smithi006.odexej (mgr.14176) 268 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:33.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:33 smithi156 bash[29655]: cluster 2023-11-17T05:40:32.591526+0000 mgr.smithi006.odexej (mgr.14176) 268 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:34.942 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager.smithi006 smithi006 *:9093,9094 running (5m) 15s ago 7m 13.6M - 0.23.0 ba2b418f427c 83267e6f9c69 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi006 smithi006 running (7m) 15s ago 7m 7627k - 16.2.14-171-ged3c8e92 6be2081e30e3 92c3b4e3c04b 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi156 smithi156 running (6m) 16s ago 6m 9639k - 16.2.14-171-ged3c8e92 6be2081e30e3 911644af628c 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:grafana.smithi006 smithi006 *:3000 running (6m) 15s ago 7m 43.2M - 8.3.5 dad864ee21e9 d6c67bd70528 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi006.odexej smithi006 *:9283 running (8m) 15s ago 8m 432M - 16.2.14-171-ged3c8e92 6be2081e30e3 563115fe905b 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi156.fphsnp smithi156 *:8443,9283 running (6m) 16s ago 6m 377M - 16.2.14-171-ged3c8e92 6be2081e30e3 36bc3689b99e 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi006 smithi006 running (8m) 15s ago 9m 42.7M 2048M 16.2.14-171-ged3c8e92 6be2081e30e3 fadc2021c8d9 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi156 smithi156 running (6m) 16s ago 6m 36.2M 2048M 16.2.14-171-ged3c8e92 6be2081e30e3 91048b84e9d5 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi006 smithi006 *:9100 running (7m) 15s ago 7m 11.9M - 1.3.1 1dbe0e931976 a6f1ad0ce93f 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi156 smithi156 *:9100 running (5m) 16s ago 6m 11.8M - 1.3.1 1dbe0e931976 720d77b63a94 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi156 running (4m) 16s ago 4m 37.4M 4041M 16.2.14-171-ged3c8e92 6be2081e30e3 e280e9ccc5db 2023-11-17T05:40:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (4m) 15s ago 4m 34.5M 3274M 16.2.14-171-ged3c8e92 6be2081e30e3 b757b0638ca6 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi156 running (4m) 16s ago 4m 34.8M 4041M 16.2.14-171-ged3c8e92 6be2081e30e3 e978fa50f36d 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi006 running (4m) 15s ago 4m 34.7M 3274M 16.2.14-171-ged3c8e92 6be2081e30e3 0285bc130b27 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi156 running (4m) 16s ago 4m 35.1M 4041M 16.2.14-171-ged3c8e92 6be2081e30e3 786ca26d7142 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi006 running (4m) 15s ago 4m 34.6M 3274M 16.2.14-171-ged3c8e92 6be2081e30e3 f09873b7db57 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 smithi156 running (4m) 16s ago 4m 34.8M 4041M 16.2.14-171-ged3c8e92 6be2081e30e3 5872f5cc7935 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 smithi006 running (4m) 15s ago 4m 35.2M 3274M 16.2.14-171-ged3c8e92 6be2081e30e3 43a73720a5e1 2023-11-17T05:40:34.944 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus.smithi006 smithi006 *:9095 running (5m) 15s ago 6m 54.9M - 2.33.4 514e6a882f6e 55f817e2693d 2023-11-17T05:40:35.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-17T05:40:35.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:35 smithi006 bash[26022]: cluster 2023-11-17T05:40:34.591953+0000 mgr.smithi006.odexej (mgr.14176) 269 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:35.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:35 smithi006 bash[26022]: audit 2023-11-17T05:40:34.915678+0000 mgr.smithi006.odexej (mgr.14176) 270 : audit [DBG] from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:35.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:35 smithi156 bash[29655]: cluster 2023-11-17T05:40:34.591953+0000 mgr.smithi006.odexej (mgr.14176) 269 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:35.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:35 smithi156 bash[29655]: audit 2023-11-17T05:40:34.915678+0000 mgr.smithi006.odexej (mgr.14176) 270 : audit [DBG] from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:37.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:37 smithi006 bash[26022]: cluster 2023-11-17T05:40:36.592656+0000 mgr.smithi006.odexej (mgr.14176) 271 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:37.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:37 smithi156 bash[29655]: cluster 2023-11-17T05:40:36.592656+0000 mgr.smithi006.odexej (mgr.14176) 271 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager ?:9093,9094 1/1 19s ago 8m count:1 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:crash 2/2 19s ago 8m * 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:grafana ?:3000 1/1 19s ago 8m count:1 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 2/2 19s ago 8m count:2 2023-11-17T05:40:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:mon 2/2 19s ago 6m smithi006:172.21.15.6=smithi006;smithi156:172.21.15.156=smithi156;count:2 2023-11-17T05:40:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter ?:9100 2/2 19s ago 8m * 2023-11-17T05:40:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:osd.all-available-devices 8 19s ago 5m * 2023-11-17T05:40:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus ?:9095 1/1 19s ago 8m count:1 2023-11-17T05:40:39.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-17T05:40:39.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:39 smithi006 bash[26022]: audit 2023-11-17T05:40:38.530474+0000 mgr.smithi006.odexej (mgr.14176) 272 : audit [DBG] from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:39.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:39 smithi006 bash[26022]: cluster 2023-11-17T05:40:38.593214+0000 mgr.smithi006.odexej (mgr.14176) 273 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:39.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:39 smithi156 bash[29655]: audit 2023-11-17T05:40:38.530474+0000 mgr.smithi006.odexej (mgr.14176) 272 : audit [DBG] from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:39.998 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:39 smithi156 bash[29655]: cluster 2023-11-17T05:40:38.593214+0000 mgr.smithi006.odexej (mgr.14176) 273 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:41.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:41 smithi006 bash[26022]: cluster 2023-11-17T05:40:40.593766+0000 mgr.smithi006.odexej (mgr.14176) 274 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:41 smithi156 bash[29655]: cluster 2023-11-17T05:40:40.593766+0000 mgr.smithi006.odexej (mgr.14176) 274 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:42.185 INFO:teuthology.orchestra.run.smithi006.stdout:HOST ADDR LABELS STATUS 2023-11-17T05:40:42.185 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 172.21.15.6 2023-11-17T05:40:42.185 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 172.21.15.156 2023-11-17T05:40:42.185 INFO:teuthology.orchestra.run.smithi006.stdout:2 hosts in cluster 2023-11-17T05:40:42.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-17T05:40:43.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:43 smithi006 bash[26022]: audit 2023-11-17T05:40:42.180243+0000 mgr.smithi006.odexej (mgr.14176) 275 : audit [DBG] from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:43.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:43 smithi006 bash[26022]: cluster 2023-11-17T05:40:42.594339+0000 mgr.smithi006.odexej (mgr.14176) 276 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:43.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:43 smithi156 bash[29655]: audit 2023-11-17T05:40:42.180243+0000 mgr.smithi006.odexej (mgr.14176) 275 : audit [DBG] from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:43.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:43 smithi156 bash[29655]: cluster 2023-11-17T05:40:42.594339+0000 mgr.smithi006.odexej (mgr.14176) 276 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:45.751 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-17T05:40:45.753 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme0n1 ssd _CVFT5330000H400BGN 372G 4m ago LVM detected, locked 2023-11-17T05:40:45.754 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme1n1 ssd _b9ec99303a4fe1b1 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.754 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme2n1 ssd _f7ff5a34bd09e3d4 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.755 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme3n1 ssd _ac7d093a0a4c141b 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.756 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme4n1 ssd _55e1634ade73856b 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.756 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme0n1 ssd _PHMB74760040480DGN 447G 4m ago LVM detected, locked 2023-11-17T05:40:45.756 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme1n1 ssd _52956b7a9abd9528 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.756 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme2n1 ssd _55a86bf49532c310 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.756 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme3n1 ssd _76b4d1da7df5bd44 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.757 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme4n1 ssd _bd01f4b66a9c929e 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:45.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:45 smithi156 bash[29655]: cluster 2023-11-17T05:40:44.595004+0000 mgr.smithi006.odexej (mgr.14176) 277 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:46.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:45 smithi006 bash[26022]: cluster 2023-11-17T05:40:44.595004+0000 mgr.smithi006.odexej (mgr.14176) 277 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:46.307 INFO:teuthology.run_tasks:Running task vip... 2023-11-17T05:40:46.324 INFO:tasks.vip:Allocating static IPs for each host... 2023-11-17T05:40:46.324 INFO:tasks.vip:peername 172.21.15.6 2023-11-17T05:40:46.328 INFO:tasks.vip:172.21.15.6 in 172.21.0.0/20, pos 3845 2023-11-17T05:40:46.335 INFO:tasks.vip:smithi006.front.sepia.ceph.com static 10.0.15.6, vnet 10.0.0.0/16 2023-11-17T05:40:46.335 INFO:tasks.vip:VIPs are {map(str, vips)} 2023-11-17T05:40:46.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip route ls 2023-11-17T05:40:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:default via 172.21.15.254 dev ens1f0 2023-11-17T05:40:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T05:40:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 2023-11-17T05:40:46.354 INFO:tasks.vip:Configuring 10.0.15.6 on smithi006.front.sepia.ceph.com iface ens1f0... 2023-11-17T05:40:46.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 10.0.15.6/16 dev ens1f0 2023-11-17T05:40:46.414 INFO:tasks.vip:peername 172.21.15.156 2023-11-17T05:40:46.416 INFO:tasks.vip:172.21.15.156 in 172.21.0.0/20, pos 3995 2023-11-17T05:40:46.423 INFO:tasks.vip:smithi156.front.sepia.ceph.com static 10.0.15.156, vnet 10.0.0.0/16 2023-11-17T05:40:46.423 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip route ls 2023-11-17T05:40:46.437 INFO:teuthology.orchestra.run.smithi156.stdout:default via 172.21.15.254 dev enp3s0f1 2023-11-17T05:40:46.437 INFO:teuthology.orchestra.run.smithi156.stdout:default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.156 metric 100 2023-11-17T05:40:46.437 INFO:teuthology.orchestra.run.smithi156.stdout:172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T05:40:46.437 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.156 2023-11-17T05:40:46.437 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.156 metric 100 2023-11-17T05:40:46.438 INFO:tasks.vip:Configuring 10.0.15.156 on smithi156.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T05:40:46.438 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr add 10.0.15.156/16 dev enp3s0f1 2023-11-17T05:40:46.498 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T05:40:46.514 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T05:40:46.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls --refresh' 2023-11-17T05:40:46.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:46 smithi156 bash[29655]: audit 2023-11-17T05:40:45.736930+0000 mgr.smithi006.odexej (mgr.14176) 278 : audit [DBG] from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:47.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:46 smithi006 bash[26022]: audit 2023-11-17T05:40:45.736930+0000 mgr.smithi006.odexej (mgr.14176) 278 : audit [DBG] from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:47.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:47 smithi006 bash[26022]: cluster 2023-11-17T05:40:46.595530+0000 mgr.smithi006.odexej (mgr.14176) 279 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:47.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:47 smithi156 bash[29655]: cluster 2023-11-17T05:40:46.595530+0000 mgr.smithi006.odexej (mgr.14176) 279 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme0n1 ssd _CVFT5330000H400BGN 372G 4m ago LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme1n1 ssd _b9ec99303a4fe1b1 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme2n1 ssd _f7ff5a34bd09e3d4 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme3n1 ssd _ac7d093a0a4c141b 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme4n1 ssd _55e1634ade73856b 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme0n1 ssd _PHMB74760040480DGN 447G 4m ago LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme1n1 ssd _52956b7a9abd9528 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme2n1 ssd _55a86bf49532c310 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme3n1 ssd _76b4d1da7df5bd44 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.631 INFO:teuthology.orchestra.run.smithi006.stdout:smithi156 /dev/nvme4n1 ssd _bd01f4b66a9c929e 89.4G 4m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T05:40:49.931 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:49 smithi156 bash[29655]: cluster 2023-11-17T05:40:48.596081+0000 mgr.smithi006.odexej (mgr.14176) 280 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:49.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:49 smithi156 bash[29655]: audit 2023-11-17T05:40:49.655062+0000 mon.smithi006 (mon.0) 608 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:40:50.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:49 smithi006 bash[26022]: cluster 2023-11-17T05:40:48.596081+0000 mgr.smithi006.odexej (mgr.14176) 280 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:50.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:49 smithi006 bash[26022]: audit 2023-11-17T05:40:49.655062+0000 mon.smithi006 (mon.0) 608 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:40:50.247 INFO:teuthology.run_tasks:Running task vip.exec... 2023-11-17T05:40:50.258 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T05:40:50.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2023-11-17T05:40:50.276 INFO:teuthology.orchestra.run.smithi006.stderr:+ systemctl stop nfs-server 2023-11-17T05:40:50.296 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi156.front.sepia.ceph.com 2023-11-17T05:40:50.296 DEBUG:teuthology.orchestra.run.smithi156:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2023-11-17T05:40:50.311 INFO:teuthology.orchestra.run.smithi156.stderr:+ systemctl stop nfs-server 2023-11-17T05:40:50.335 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T05:40:50.354 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T05:40:50.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph fs volume create foofs' 2023-11-17T05:40:51.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:50 smithi006 bash[26022]: audit 2023-11-17T05:40:49.618924+0000 mgr.smithi006.odexej (mgr.14176) 281 : audit [DBG] from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:51.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:50 smithi156 bash[29655]: audit 2023-11-17T05:40:49.618924+0000 mgr.smithi006.odexej (mgr.14176) 281 : audit [DBG] from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:51.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:51 smithi006 bash[26022]: cluster 2023-11-17T05:40:50.596512+0000 mgr.smithi006.odexej (mgr.14176) 282 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:52.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:51 smithi156 bash[29655]: cluster 2023-11-17T05:40:50.596512+0000 mgr.smithi006.odexej (mgr.14176) 282 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:54.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:54 smithi006 bash[26022]: cluster 2023-11-17T05:40:52.597110+0000 mgr.smithi006.odexej (mgr.14176) 283 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:54.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:54 smithi156 bash[29655]: cluster 2023-11-17T05:40:52.597110+0000 mgr.smithi006.odexej (mgr.14176) 283 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:55.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: audit 2023-11-17T05:40:53.630267+0000 mgr.smithi006.odexej (mgr.14176) 284 : audit [DBG] from='client.14480 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: audit 2023-11-17T05:40:53.631239+0000 mon.smithi006 (mon.0) 609 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: audit 2023-11-17T05:40:54.143141+0000 mon.smithi006 (mon.0) 610 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: cluster 2023-11-17T05:40:54.143460+0000 mon.smithi006 (mon.0) 611 : cluster [DBG] osdmap e37: 8 total, 8 up, 8 in 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: audit 2023-11-17T05:40:54.145550+0000 mon.smithi006 (mon.0) 612 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: cluster 2023-11-17T05:40:54.597805+0000 mgr.smithi006.odexej (mgr.14176) 285 : cluster [DBG] pgmap v246: 33 pgs: 15 creating+peering, 1 active+clean, 17 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:55.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:55 smithi006 bash[26022]: debug 2023-11-17T05:40:55.157+0000 7fe97f302700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T05:40:55.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: audit 2023-11-17T05:40:53.630267+0000 mgr.smithi006.odexej (mgr.14176) 284 : audit [DBG] from='client.14480 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:40:55.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: audit 2023-11-17T05:40:53.631239+0000 mon.smithi006 (mon.0) 609 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2023-11-17T05:40:55.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: audit 2023-11-17T05:40:54.143141+0000 mon.smithi006 (mon.0) 610 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2023-11-17T05:40:55.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: cluster 2023-11-17T05:40:54.143460+0000 mon.smithi006 (mon.0) 611 : cluster [DBG] osdmap e37: 8 total, 8 up, 8 in 2023-11-17T05:40:55.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: audit 2023-11-17T05:40:54.145550+0000 mon.smithi006 (mon.0) 612 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2023-11-17T05:40:55.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:55 smithi156 bash[29655]: cluster 2023-11-17T05:40:54.597805+0000 mgr.smithi006.odexej (mgr.14176) 285 : cluster [DBG] pgmap v246: 33 pgs: 15 creating+peering, 1 active+clean, 17 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:55.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph nfs cluster create foo --ingress --virtual-ip 10.0.31.6/16 --port 2999' 2023-11-17T05:40:56.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: audit 2023-11-17T05:40:55.151797+0000 mon.smithi006 (mon.0) 613 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cluster 2023-11-17T05:40:55.152008+0000 mon.smithi006 (mon.0) 614 : cluster [DBG] osdmap e38: 8 total, 8 up, 8 in 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: audit 2023-11-17T05:40:55.160792+0000 mon.smithi006 (mon.0) 615 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cluster 2023-11-17T05:40:55.161774+0000 mon.smithi006 (mon.0) 616 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cluster 2023-11-17T05:40:55.161827+0000 mon.smithi006 (mon.0) 617 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cluster 2023-11-17T05:40:55.171635+0000 mon.smithi006 (mon.0) 618 : cluster [DBG] osdmap e39: 8 total, 8 up, 8 in 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: audit 2023-11-17T05:40:55.172634+0000 mon.smithi006 (mon.0) 619 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cluster 2023-11-17T05:40:55.172867+0000 mon.smithi006 (mon.0) 620 : cluster [DBG] fsmap foofs:0 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: cephadm 2023-11-17T05:40:55.178149+0000 mgr.smithi006.odexej (mgr.14176) 286 : cephadm [INF] Saving service mds.foofs spec with placement count:2 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: audit 2023-11-17T05:40:55.183710+0000 mon.smithi006 (mon.0) 621 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:56 smithi006 bash[26022]: audit 2023-11-17T05:40:55.289361+0000 mon.smithi006 (mon.0) 622 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: audit 2023-11-17T05:40:55.151797+0000 mon.smithi006 (mon.0) 613 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2023-11-17T05:40:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cluster 2023-11-17T05:40:55.152008+0000 mon.smithi006 (mon.0) 614 : cluster [DBG] osdmap e38: 8 total, 8 up, 8 in 2023-11-17T05:40:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: audit 2023-11-17T05:40:55.160792+0000 mon.smithi006 (mon.0) 615 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2023-11-17T05:40:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cluster 2023-11-17T05:40:55.161774+0000 mon.smithi006 (mon.0) 616 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T05:40:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cluster 2023-11-17T05:40:55.161827+0000 mon.smithi006 (mon.0) 617 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cluster 2023-11-17T05:40:55.171635+0000 mon.smithi006 (mon.0) 618 : cluster [DBG] osdmap e39: 8 total, 8 up, 8 in 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: audit 2023-11-17T05:40:55.172634+0000 mon.smithi006 (mon.0) 619 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cluster 2023-11-17T05:40:55.172867+0000 mon.smithi006 (mon.0) 620 : cluster [DBG] fsmap foofs:0 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: cephadm 2023-11-17T05:40:55.178149+0000 mgr.smithi006.odexej (mgr.14176) 286 : cephadm [INF] Saving service mds.foofs spec with placement count:2 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: audit 2023-11-17T05:40:55.183710+0000 mon.smithi006 (mon.0) 621 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:56 smithi156 bash[29655]: audit 2023-11-17T05:40:55.289361+0000 mon.smithi006 (mon.0) 622 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:57.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:57 smithi006 bash[26022]: cluster 2023-11-17T05:40:56.171044+0000 mon.smithi006 (mon.0) 623 : cluster [DBG] osdmap e40: 8 total, 8 up, 8 in 2023-11-17T05:40:57.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:57 smithi006 bash[26022]: cluster 2023-11-17T05:40:56.598554+0000 mgr.smithi006.odexej (mgr.14176) 287 : cluster [DBG] pgmap v250: 65 pgs: 5 active+clean, 19 creating+peering, 41 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:57.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:57 smithi156 bash[29655]: cluster 2023-11-17T05:40:56.171044+0000 mon.smithi006 (mon.0) 623 : cluster [DBG] osdmap e40: 8 total, 8 up, 8 in 2023-11-17T05:40:57.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:57 smithi156 bash[29655]: cluster 2023-11-17T05:40:56.598554+0000 mgr.smithi006.odexej (mgr.14176) 287 : cluster [DBG] pgmap v250: 65 pgs: 5 active+clean, 19 creating+peering, 41 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:40:58.627 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:58 smithi006 bash[26022]: cluster 2023-11-17T05:40:57.174224+0000 mon.smithi006 (mon.0) 624 : cluster [DBG] osdmap e41: 8 total, 8 up, 8 in 2023-11-17T05:40:58.627 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:58 smithi006 bash[26022]: audit 2023-11-17T05:40:57.190843+0000 mon.smithi006 (mon.0) 625 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:58.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:58 smithi156 bash[29655]: cluster 2023-11-17T05:40:57.174224+0000 mon.smithi006 (mon.0) 624 : cluster [DBG] osdmap e41: 8 total, 8 up, 8 in 2023-11-17T05:40:58.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:58 smithi156 bash[29655]: audit 2023-11-17T05:40:57.190843+0000 mon.smithi006 (mon.0) 625 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:40:59.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:40:59 smithi006 bash[26022]: cluster 2023-11-17T05:40:58.599127+0000 mgr.smithi006.odexej (mgr.14176) 288 : cluster [DBG] pgmap v252: 65 pgs: 26 active+clean, 19 creating+peering, 20 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.850 INFO:teuthology.orchestra.run.smithi006.stdout:NFS Cluster Created Successfully 2023-11-17T05:42:21.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:00 smithi006 bash[26022]: audit 2023-11-17T05:40:59.103214+0000 mgr.smithi006.odexej (mgr.14176) 289 : audit [DBG] from='client.14484 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "ingress": true, "virtual_ip": "10.0.31.6/16", "port": 2999, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:42:21.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:00 smithi006 bash[26022]: audit 2023-11-17T05:40:59.106572+0000 mon.smithi006 (mon.0) 626 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2023-11-17T05:42:21.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:00 smithi006 bash[26022]: audit 2023-11-17T05:40:59.620100+0000 mon.smithi006 (mon.0) 627 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2023-11-17T05:42:21.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:00 smithi006 bash[26022]: cluster 2023-11-17T05:40:59.620224+0000 mon.smithi006 (mon.0) 628 : cluster [DBG] osdmap e42: 8 total, 8 up, 8 in 2023-11-17T05:42:21.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:00 smithi006 bash[26022]: audit 2023-11-17T05:40:59.627149+0000 mon.smithi006 (mon.0) 629 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2023-11-17T05:42:21.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2023-11-17T05:42:21.957 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:40:59 smithi156 bash[29655]: cluster 2023-11-17T05:40:58.599127+0000 mgr.smithi006.odexej (mgr.14176) 288 : cluster [DBG] pgmap v252: 65 pgs: 26 active+clean, 19 creating+peering, 20 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.957 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:00 smithi156 bash[29655]: audit 2023-11-17T05:40:59.103214+0000 mgr.smithi006.odexej (mgr.14176) 289 : audit [DBG] from='client.14484 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "ingress": true, "virtual_ip": "10.0.31.6/16", "port": 2999, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:42:21.957 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:00 smithi156 bash[29655]: audit 2023-11-17T05:40:59.106572+0000 mon.smithi006 (mon.0) 626 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:00 smithi156 bash[29655]: audit 2023-11-17T05:40:59.620100+0000 mon.smithi006 (mon.0) 627 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:00 smithi156 bash[29655]: cluster 2023-11-17T05:40:59.620224+0000 mon.smithi006 (mon.0) 628 : cluster [DBG] osdmap e42: 8 total, 8 up, 8 in 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:00 smithi156 bash[29655]: audit 2023-11-17T05:40:59.627149+0000 mon.smithi006 (mon.0) 629 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: cluster 2023-11-17T05:41:00.599857+0000 mgr.smithi006.odexej (mgr.14176) 290 : cluster [DBG] pgmap v254: 97 pgs: 1 creating+peering, 31 unknown, 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: audit 2023-11-17T05:41:00.621983+0000 mon.smithi006 (mon.0) 630 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2023-11-17T05:42:21.958 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: cluster 2023-11-17T05:41:00.622097+0000 mon.smithi006 (mon.0) 631 : cluster [DBG] osdmap e43: 8 total, 8 up, 8 in 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: cephadm 2023-11-17T05:41:00.654157+0000 mgr.smithi006.odexej (mgr.14176) 291 : cephadm [INF] Saving service nfs.foo spec with placement count:1 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: audit 2023-11-17T05:41:00.657664+0000 mon.smithi006 (mon.0) 632 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: cephadm 2023-11-17T05:41:00.659406+0000 mgr.smithi006.odexej (mgr.14176) 292 : cephadm [INF] Saving service ingress.nfs.foo spec with placement count:2 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:01 smithi156 bash[29655]: audit 2023-11-17T05:41:00.662386+0000 mon.smithi006 (mon.0) 633 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:02 smithi156 bash[29655]: cluster 2023-11-17T05:41:01.625732+0000 mon.smithi006 (mon.0) 634 : cluster [DBG] osdmap e44: 8 total, 8 up, 8 in 2023-11-17T05:42:21.959 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:02 smithi156 bash[29655]: cluster 2023-11-17T05:41:01.677180+0000 mon.smithi006 (mon.0) 635 : cluster [DBG] mgrmap e19: smithi006.odexej(active, since 7m), standbys: smithi156.fphsnp 2023-11-17T05:42:21.960 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:03 smithi156 bash[29655]: cluster 2023-11-17T05:41:02.600681+0000 mgr.smithi006.odexej (mgr.14176) 293 : cluster [DBG] pgmap v257: 97 pgs: 1 creating+peering, 27 unknown, 69 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.960 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:05 smithi156 bash[29655]: cluster 2023-11-17T05:41:04.601382+0000 mgr.smithi006.odexej (mgr.14176) 294 : cluster [DBG] pgmap v258: 97 pgs: 1 creating+peering, 96 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.960 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:05 smithi156 bash[29655]: audit 2023-11-17T05:41:04.701214+0000 mon.smithi006 (mon.0) 636 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.963 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:05 smithi156 bash[29655]: audit 2023-11-17T05:41:04.835071+0000 mon.smithi006 (mon.0) 637 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.963 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:07 smithi156 bash[29655]: cluster 2023-11-17T05:41:06.602025+0000 mgr.smithi006.odexej (mgr.14176) 295 : cluster [DBG] pgmap v259: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.963 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.053481+0000 mon.smithi006 (mon.0) 638 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.963 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.055524+0000 mon.smithi006 (mon.0) 639 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.056918+0000 mon.smithi006 (mon.0) 640 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.065311+0000 mon.smithi006 (mon.0) 641 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.075905+0000 mon.smithi006 (mon.0) 642 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.093934+0000 mon.smithi006 (mon.0) 643 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.095421+0000 mon.smithi006 (mon.0) 644 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi156.gmiiei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T05:42:21.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.101572+0000 mon.smithi006 (mon.0) 645 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi156.gmiiei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T05:42:21.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: audit 2023-11-17T05:41:08.102819+0000 mon.smithi006 (mon.0) 646 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:21.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: cephadm 2023-11-17T05:41:08.104055+0000 mgr.smithi006.odexej (mgr.14176) 296 : cephadm [INF] Deploying daemon mds.foofs.smithi156.gmiiei on smithi156 2023-11-17T05:42:21.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:09 smithi156 bash[29655]: cluster 2023-11-17T05:41:08.602965+0000 mgr.smithi006.odexej (mgr.14176) 297 : cluster [DBG] pgmap v260: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: audit 2023-11-17T05:41:10.377132+0000 mon.smithi006 (mon.0) 647 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: audit 2023-11-17T05:41:10.379890+0000 mon.smithi006 (mon.0) 648 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi006.rwbcnj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T05:42:21.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: audit 2023-11-17T05:41:10.386849+0000 mon.smithi006 (mon.0) 649 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi006.rwbcnj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T05:42:21.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: audit 2023-11-17T05:41:10.388583+0000 mon.smithi006 (mon.0) 650 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:21.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: cephadm 2023-11-17T05:41:10.390006+0000 mgr.smithi006.odexej (mgr.14176) 298 : cephadm [INF] Deploying daemon mds.foofs.smithi006.rwbcnj on smithi006 2023-11-17T05:42:21.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:11 smithi156 bash[29655]: cluster 2023-11-17T05:41:10.604409+0000 mgr.smithi006.odexej (mgr.14176) 299 : cluster [DBG] pgmap v261: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.390734+0000 mon.smithi006 (mon.0) 651 : cluster [DBG] mds.? [v2:172.21.15.156:6832/845746766,v1:172.21.15.156:6833/845746766] up:boot 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.390868+0000 mon.smithi006 (mon.0) 652 : cluster [INF] daemon mds.foofs.smithi156.gmiiei assigned to filesystem foofs as rank 0 (now has 1 ranks) 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.391056+0000 mon.smithi006 (mon.0) 653 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.391109+0000 mon.smithi006 (mon.0) 654 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.391141+0000 mon.smithi006 (mon.0) 655 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:21.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.391549+0000 mon.smithi006 (mon.0) 656 : cluster [DBG] fsmap foofs:0 1 up:standby 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.391732+0000 mon.smithi006 (mon.0) 657 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi156.gmiiei"}]: dispatch 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.397563+0000 mon.smithi006 (mon.0) 658 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:creating} 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.415051+0000 mon.smithi006 (mon.0) 659 : cluster [INF] daemon mds.foofs.smithi156.gmiiei is now active in filesystem foofs as rank 0 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.596662+0000 mon.smithi006 (mon.0) 660 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cluster 2023-11-17T05:41:12.605140+0000 mgr.smithi006.odexej (mgr.14176) 300 : cluster [DBG] pgmap v262: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:21.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.605257+0000 mon.smithi006 (mon.0) 661 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cephadm 2023-11-17T05:41:12.635652+0000 mgr.smithi006.odexej (mgr.14176) 301 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi006.uasswy 2023-11-17T05:42:21.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.636106+0000 mon.smithi006 (mon.0) 662 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T05:42:21.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.642261+0000 mon.smithi006 (mon.0) 663 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T05:42:21.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cephadm 2023-11-17T05:41:12.643025+0000 mgr.smithi006.odexej (mgr.14176) 302 : cephadm [INF] Ensuring nfs.foo.0 is in the ganesha grace table 2023-11-17T05:42:21.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.643350+0000 mon.smithi006 (mon.0) 664 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.649091+0000 mon.smithi006 (mon.0) 665 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.650591+0000 mon.smithi006 (mon.0) 666 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.706051+0000 mon.smithi006 (mon.0) 667 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.710097+0000 mon.smithi006 (mon.0) 668 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cephadm 2023-11-17T05:41:12.775035+0000 mgr.smithi006.odexej (mgr.14176) 303 : cephadm [INF] Rados config object exists: conf-nfs.foo 2023-11-17T05:42:21.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cephadm 2023-11-17T05:41:12.775308+0000 mgr.smithi006.odexej (mgr.14176) 304 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi006.uasswy-rgw 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.776111+0000 mon.smithi006 (mon.0) 669 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.784839+0000 mon.smithi006 (mon.0) 670 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: audit 2023-11-17T05:41:12.795724+0000 mon.smithi006 (mon.0) 671 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:13 smithi156 bash[29655]: cephadm 2023-11-17T05:41:12.797779+0000 mgr.smithi006.odexej (mgr.14176) 305 : cephadm [INF] Deploying daemon nfs.foo.0.0.smithi006.uasswy on smithi006 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:14 smithi156 bash[29655]: cluster 2023-11-17T05:41:13.402408+0000 mon.smithi006 (mon.0) 672 : cluster [DBG] mds.? [v2:172.21.15.156:6832/845746766,v1:172.21.15.156:6833/845746766] up:active 2023-11-17T05:42:21.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:14 smithi156 bash[29655]: cluster 2023-11-17T05:41:13.402512+0000 mon.smithi006 (mon.0) 673 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 2023-11-17T05:42:21.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: cluster 2023-11-17T05:41:14.606578+0000 mgr.smithi006.odexej (mgr.14176) 306 : cluster [DBG] pgmap v263: 97 pgs: 97 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-17T05:42:21.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: cluster 2023-11-17T05:41:14.613621+0000 mon.smithi006 (mon.0) 674 : cluster [DBG] mds.? [v2:172.21.15.6:6834/3961950015,v1:172.21.15.6:6835/3961950015] up:boot 2023-11-17T05:42:21.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: cluster 2023-11-17T05:41:14.614394+0000 mon.smithi006 (mon.0) 675 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 1 up:standby 2023-11-17T05:42:21.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: audit 2023-11-17T05:41:14.614639+0000 mon.smithi006 (mon.0) 676 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi006.rwbcnj"}]: dispatch 2023-11-17T05:42:21.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: cluster 2023-11-17T05:41:14.621360+0000 mon.smithi006 (mon.0) 677 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 1 up:standby 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: audit 2023-11-17T05:41:15.028432+0000 mon.smithi006 (mon.0) 678 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: audit 2023-11-17T05:41:15.254372+0000 mon.smithi006 (mon.0) 679 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:15 smithi156 bash[29655]: audit 2023-11-17T05:41:15.264265+0000 mon.smithi006 (mon.0) 680 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:16 smithi156 bash[29655]: cephadm 2023-11-17T05:41:15.282215+0000 mgr.smithi006.odexej (mgr.14176) 307 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.uylhhq on smithi156 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:17 smithi156 bash[29655]: cluster 2023-11-17T05:41:16.607440+0000 mgr.smithi006.odexej (mgr.14176) 308 : cluster [DBG] pgmap v264: 97 pgs: 97 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1023 B/s wr, 3 op/s 2023-11-17T05:42:21.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:19 smithi156 bash[29655]: cluster 2023-11-17T05:41:18.608420+0000 mgr.smithi006.odexej (mgr.14176) 309 : cluster [DBG] pgmap v265: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:19 smithi156 bash[29655]: audit 2023-11-17T05:41:18.989070+0000 mon.smithi006 (mon.0) 681 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.uylhhq"}]: dispatch 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: cephadm 2023-11-17T05:41:18.987793+0000 mgr.smithi006.odexej (mgr.14176) 310 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:21.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:21.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:21.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:21.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:21.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:21.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:21.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:21.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:21.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:21.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:21.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:21.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:22.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:22.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:22.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.030 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: cephadm 2023-11-17T05:41:18.988395+0000 mgr.smithi006.odexej (mgr.14176) 311 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: cephadm 2023-11-17T05:41:18.990904+0000 mgr.smithi006.odexej (mgr.14176) 312 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.uylhhq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:22.031 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: cephadm 2023-11-17T05:41:18.996324+0000 mgr.smithi006.odexej (mgr.14176) 313 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.iyogat on smithi006 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: audit 2023-11-17T05:41:19.701781+0000 mon.smithi006 (mon.0) 682 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:20 smithi156 bash[29655]: audit 2023-11-17T05:41:20.039796+0000 mon.smithi006 (mon.0) 683 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:21 smithi156 bash[29655]: cluster 2023-11-17T05:41:20.609550+0000 mgr.smithi006.odexej (mgr.14176) 314 : cluster [DBG] pgmap v266: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: cluster 2023-11-17T05:41:22.610350+0000 mgr.smithi006.odexej (mgr.14176) 315 : cluster [DBG] pgmap v267: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: cephadm 2023-11-17T05:41:22.945695+0000 mgr.smithi006.odexej (mgr.14176) 316 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.032 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.033 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.177 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: cephadm 2023-11-17T05:41:22.946273+0000 mgr.smithi006.odexej (mgr.14176) 317 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: audit 2023-11-17T05:41:22.947034+0000 mon.smithi006 (mon.0) 684 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.iyogat"}]: dispatch 2023-11-17T05:42:22.178 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: audit 2023-11-17T05:41:22.958090+0000 mon.smithi006 (mon.0) 685 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:23 smithi156 bash[29655]: audit 2023-11-17T05:41:22.962407+0000 mon.smithi006 (mon.0) 686 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: cephadm 2023-11-17T05:41:22.948812+0000 mgr.smithi006.odexej (mgr.14176) 318 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.iyogat on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:22.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: cephadm 2023-11-17T05:41:22.958579+0000 mgr.smithi006.odexej (mgr.14176) 319 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.wxztob on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: cephadm 2023-11-17T05:41:22.961034+0000 mgr.smithi006.odexej (mgr.14176) 320 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ticlht on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: cluster 2023-11-17T05:41:22.962109+0000 mgr.smithi006.odexej (mgr.14176) 321 : cluster [DBG] pgmap v268: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:24 smithi156 bash[29655]: cluster 2023-11-17T05:41:23.955933+0000 mon.smithi006 (mon.0) 687 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:26 smithi156 bash[29655]: cluster 2023-11-17T05:41:24.963457+0000 mgr.smithi006.odexej (mgr.14176) 322 : cluster [DBG] pgmap v269: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 B/s wr, 2 op/s 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:26 smithi156 bash[29655]: audit 2023-11-17T05:41:25.052122+0000 mon.smithi006 (mon.0) 688 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:28 smithi156 bash[29655]: cluster 2023-11-17T05:41:26.964437+0000 mgr.smithi006.odexej (mgr.14176) 323 : cluster [DBG] pgmap v270: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 B/s wr, 2 op/s 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:29 smithi156 bash[29655]: audit 2023-11-17T05:41:28.645911+0000 mon.smithi006 (mon.0) 689 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:29 smithi156 bash[29655]: audit 2023-11-17T05:41:29.488253+0000 mon.smithi006 (mon.0) 690 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: cluster 2023-11-17T05:41:28.965535+0000 mgr.smithi006.odexej (mgr.14176) 324 : cluster [DBG] pgmap v271: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 B/s wr, 0 op/s 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: audit 2023-11-17T05:41:29.786531+0000 mon.smithi006 (mon.0) 691 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:22.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: audit 2023-11-17T05:41:29.788099+0000 mon.smithi006 (mon.0) 692 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: audit 2023-11-17T05:41:29.797618+0000 mon.smithi006 (mon.0) 693 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: cluster 2023-11-17T05:41:29.798817+0000 mgr.smithi006.odexej (mgr.14176) 325 : cluster [DBG] pgmap v272: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 111 B/s rd, 0 op/s 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: audit 2023-11-17T05:41:29.804010+0000 mon.smithi006 (mon.0) 694 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: audit 2023-11-17T05:41:29.820535+0000 mon.smithi006 (mon.0) 695 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:30 smithi156 bash[29655]: cephadm 2023-11-17T05:41:29.825359+0000 mgr.smithi006.odexej (mgr.14176) 326 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dnwbpo on smithi156 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:31 smithi156 bash[29655]: cluster 2023-11-17T05:41:30.794210+0000 mon.smithi006 (mon.0) 696 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:31 smithi156 bash[29655]: cluster 2023-11-17T05:41:30.794276+0000 mon.smithi006 (mon.0) 697 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:32 smithi156 bash[29655]: cluster 2023-11-17T05:41:31.799598+0000 mgr.smithi006.odexej (mgr.14176) 327 : cluster [DBG] pgmap v273: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:33 smithi156 bash[29655]: audit 2023-11-17T05:41:33.536046+0000 mon.smithi006 (mon.0) 698 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dnwbpo"}]: dispatch 2023-11-17T05:42:22.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: cephadm 2023-11-17T05:41:33.534484+0000 mgr.smithi006.odexej (mgr.14176) 328 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.318 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.320 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.321 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: cephadm 2023-11-17T05:41:33.535089+0000 mgr.smithi006.odexej (mgr.14176) 329 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: cephadm 2023-11-17T05:41:33.537712+0000 mgr.smithi006.odexej (mgr.14176) 330 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dnwbpo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.322 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.323 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: cephadm 2023-11-17T05:41:33.541038+0000 mgr.smithi006.odexej (mgr.14176) 331 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.kyltxj on smithi006 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:34 smithi156 bash[29655]: cluster 2023-11-17T05:41:33.800237+0000 mgr.smithi006.odexej (mgr.14176) 332 : cluster [DBG] pgmap v274: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:36 smithi156 bash[29655]: cluster 2023-11-17T05:41:35.801305+0000 mgr.smithi006.odexej (mgr.14176) 333 : cluster [DBG] pgmap v275: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:37 smithi156 bash[29655]: audit 2023-11-17T05:41:37.268055+0000 mon.smithi006 (mon.0) 699 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.kyltxj"}]: dispatch 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:37 smithi156 bash[29655]: audit 2023-11-17T05:41:37.279759+0000 mon.smithi006 (mon.0) 700 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cephadm 2023-11-17T05:41:37.266820+0000 mgr.smithi006.odexej (mgr.14176) 334 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:22.324 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.325 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.325 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.325 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.325 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.325 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:22.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:22.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:22.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.525 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.525 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.526 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cephadm 2023-11-17T05:41:37.267245+0000 mgr.smithi006.odexej (mgr.14176) 335 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cephadm 2023-11-17T05:41:37.269525+0000 mgr.smithi006.odexej (mgr.14176) 336 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.kyltxj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:22.527 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.528 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.528 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:22.528 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:22.528 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.528 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.529 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.529 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cephadm 2023-11-17T05:41:37.272830+0000 mgr.smithi006.odexej (mgr.14176) 337 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vmmwrr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cephadm 2023-11-17T05:41:37.275352+0000 mgr.smithi006.odexej (mgr.14176) 338 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cluster 2023-11-17T05:41:37.276759+0000 mgr.smithi006.odexej (mgr.14176) 339 : cluster [DBG] pgmap v276: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:38 smithi156 bash[29655]: cluster 2023-11-17T05:41:37.858504+0000 mon.smithi006 (mon.0) 701 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:22.530 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:40 smithi156 bash[29655]: cluster 2023-11-17T05:41:39.278117+0000 mgr.smithi006.odexej (mgr.14176) 340 : cluster [DBG] pgmap v277: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:42:22.531 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:40 smithi156 bash[29655]: audit 2023-11-17T05:41:40.072676+0000 mon.smithi006 (mon.0) 702 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:42 smithi156 bash[29655]: cluster 2023-11-17T05:41:41.279257+0000 mgr.smithi006.odexej (mgr.14176) 341 : cluster [DBG] pgmap v278: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:42.984191+0000 mon.smithi006 (mon.0) 703 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: cluster 2023-11-17T05:41:43.280383+0000 mgr.smithi006.odexej (mgr.14176) 342 : cluster [DBG] pgmap v279: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.770942+0000 mon.smithi006 (mon.0) 704 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.772729+0000 mon.smithi006 (mon.0) 705 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.774526+0000 mon.smithi006 (mon.0) 706 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.784531+0000 mon.smithi006 (mon.0) 707 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: cluster 2023-11-17T05:41:43.785722+0000 mgr.smithi006.odexej (mgr.14176) 343 : cluster [DBG] pgmap v280: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.793473+0000 mon.smithi006 (mon.0) 708 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:22.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: audit 2023-11-17T05:41:43.813746+0000 mon.smithi006 (mon.0) 709 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:43 smithi156 bash[29655]: cephadm 2023-11-17T05:41:43.818239+0000 mgr.smithi006.odexej (mgr.14176) 344 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xaaadw on smithi156 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:44 smithi156 bash[29655]: cluster 2023-11-17T05:41:43.980810+0000 mon.smithi006 (mon.0) 710 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:44 smithi156 bash[29655]: cluster 2023-11-17T05:41:43.980897+0000 mon.smithi006 (mon.0) 711 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:46 smithi156 bash[29655]: cluster 2023-11-17T05:41:45.786785+0000 mgr.smithi006.odexej (mgr.14176) 345 : cluster [DBG] pgmap v281: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:47 smithi156 bash[29655]: audit 2023-11-17T05:41:47.550718+0000 mon.smithi006 (mon.0) 712 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xaaadw"}]: dispatch 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: cephadm 2023-11-17T05:41:47.549525+0000 mgr.smithi006.odexej (mgr.14176) 346 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:22.559 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.560 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: cephadm 2023-11-17T05:41:47.550063+0000 mgr.smithi006.odexej (mgr.14176) 347 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: cephadm 2023-11-17T05:41:47.552522+0000 mgr.smithi006.odexej (mgr.14176) 348 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.561 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: cephadm 2023-11-17T05:41:47.557721+0000 mgr.smithi006.odexej (mgr.14176) 349 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.bnvxvk on smithi006 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:48 smithi156 bash[29655]: cluster 2023-11-17T05:41:47.787381+0000 mgr.smithi006.odexej (mgr.14176) 350 : cluster [DBG] pgmap v282: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:50 smithi156 bash[29655]: cluster 2023-11-17T05:41:49.788478+0000 mgr.smithi006.odexej (mgr.14176) 351 : cluster [DBG] pgmap v283: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cephadm 2023-11-17T05:41:51.306868+0000 mgr.smithi006.odexej (mgr.14176) 352 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.562 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.643 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.643 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.643 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.643 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.644 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.674 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.674 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.674 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cephadm 2023-11-17T05:41:51.307438+0000 mgr.smithi006.odexej (mgr.14176) 353 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: audit 2023-11-17T05:41:51.308174+0000 mon.smithi006 (mon.0) 713 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.bnvxvk"}]: dispatch 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cephadm 2023-11-17T05:41:51.309934+0000 mgr.smithi006.odexej (mgr.14176) 354 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.675 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:22.676 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cephadm 2023-11-17T05:41:51.314376+0000 mgr.smithi006.odexej (mgr.14176) 355 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cephadm 2023-11-17T05:41:51.319179+0000 mgr.smithi006.odexej (mgr.14176) 356 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:52 smithi156 bash[29655]: cluster 2023-11-17T05:41:51.320369+0000 mgr.smithi006.odexej (mgr.14176) 357 : cluster [DBG] pgmap v284: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:53 smithi156 bash[29655]: cluster 2023-11-17T05:41:52.259720+0000 mon.smithi006 (mon.0) 714 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:54 smithi156 bash[29655]: cluster 2023-11-17T05:41:53.321526+0000 mgr.smithi006.odexej (mgr.14176) 358 : cluster [DBG] pgmap v285: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:56 smithi156 bash[29655]: audit 2023-11-17T05:41:55.092689+0000 mon.smithi006 (mon.0) 715 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:56 smithi156 bash[29655]: cluster 2023-11-17T05:41:55.322845+0000 mgr.smithi006.odexej (mgr.14176) 359 : cluster [DBG] pgmap v286: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:41:58 smithi156 bash[29655]: cluster 2023-11-17T05:41:57.324004+0000 mgr.smithi006.odexej (mgr.14176) 360 : cluster [DBG] pgmap v287: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:00 smithi156 bash[29655]: cluster 2023-11-17T05:41:59.325412+0000 mgr.smithi006.odexej (mgr.14176) 361 : cluster [DBG] pgmap v288: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:02 smithi156 bash[29655]: cluster 2023-11-17T05:42:01.326453+0000 mgr.smithi006.odexej (mgr.14176) 362 : cluster [DBG] pgmap v289: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:04 smithi156 bash[29655]: cluster 2023-11-17T05:42:03.327603+0000 mgr.smithi006.odexej (mgr.14176) 363 : cluster [DBG] pgmap v290: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:22.677 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:05 smithi156 bash[29655]: audit 2023-11-17T05:42:04.703247+0000 mon.smithi006 (mon.0) 716 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.678 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:06 smithi156 bash[29655]: cluster 2023-11-17T05:42:05.328927+0000 mgr.smithi006.odexej (mgr.14176) 364 : cluster [DBG] pgmap v291: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.678 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:08 smithi156 bash[29655]: cluster 2023-11-17T05:42:07.330048+0000 mgr.smithi006.odexej (mgr.14176) 365 : cluster [DBG] pgmap v292: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.678 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:10 smithi156 bash[29655]: cluster 2023-11-17T05:42:09.331492+0000 mgr.smithi006.odexej (mgr.14176) 366 : cluster [DBG] pgmap v293: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.678 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:12 smithi156 bash[29655]: cluster 2023-11-17T05:42:11.332521+0000 mgr.smithi006.odexej (mgr.14176) 367 : cluster [DBG] pgmap v294: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.808 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:14 smithi156 bash[29655]: cluster 2023-11-17T05:42:13.333609+0000 mgr.smithi006.odexej (mgr.14176) 368 : cluster [DBG] pgmap v295: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:22.808 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:16 smithi156 bash[29655]: cluster 2023-11-17T05:42:15.334942+0000 mgr.smithi006.odexej (mgr.14176) 369 : cluster [DBG] pgmap v296: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:18 smithi156 bash[29655]: cluster 2023-11-17T05:42:17.336008+0000 mgr.smithi006.odexej (mgr.14176) 370 : cluster [DBG] pgmap v297: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.809 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:20 smithi156 bash[29655]: cluster 2023-11-17T05:42:19.337439+0000 mgr.smithi006.odexej (mgr.14176) 371 : cluster [DBG] pgmap v298: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:22 smithi156 bash[29655]: cluster 2023-11-17T05:42:21.338458+0000 mgr.smithi006.odexej (mgr.14176) 372 : cluster [DBG] pgmap v299: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:22.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: cluster 2023-11-17T05:41:00.599857+0000 mgr.smithi006.odexej (mgr.14176) 290 : cluster [DBG] pgmap v254: 97 pgs: 1 creating+peering, 31 unknown, 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: audit 2023-11-17T05:41:00.621983+0000 mon.smithi006 (mon.0) 630 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2023-11-17T05:42:22.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: cluster 2023-11-17T05:41:00.622097+0000 mon.smithi006 (mon.0) 631 : cluster [DBG] osdmap e43: 8 total, 8 up, 8 in 2023-11-17T05:42:22.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: cephadm 2023-11-17T05:41:00.654157+0000 mgr.smithi006.odexej (mgr.14176) 291 : cephadm [INF] Saving service nfs.foo spec with placement count:1 2023-11-17T05:42:22.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: audit 2023-11-17T05:41:00.657664+0000 mon.smithi006 (mon.0) 632 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: cephadm 2023-11-17T05:41:00.659406+0000 mgr.smithi006.odexej (mgr.14176) 292 : cephadm [INF] Saving service ingress.nfs.foo spec with placement count:2 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:01 smithi006 bash[26022]: audit 2023-11-17T05:41:00.662386+0000 mon.smithi006 (mon.0) 633 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:02 smithi006 bash[26022]: cluster 2023-11-17T05:41:01.625732+0000 mon.smithi006 (mon.0) 634 : cluster [DBG] osdmap e44: 8 total, 8 up, 8 in 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:02 smithi006 bash[26022]: cluster 2023-11-17T05:41:01.677180+0000 mon.smithi006 (mon.0) 635 : cluster [DBG] mgrmap e19: smithi006.odexej(active, since 7m), standbys: smithi156.fphsnp 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:03 smithi006 bash[26022]: cluster 2023-11-17T05:41:02.600681+0000 mgr.smithi006.odexej (mgr.14176) 293 : cluster [DBG] pgmap v257: 97 pgs: 1 creating+peering, 27 unknown, 69 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:05 smithi006 bash[26022]: cluster 2023-11-17T05:41:04.601382+0000 mgr.smithi006.odexej (mgr.14176) 294 : cluster [DBG] pgmap v258: 97 pgs: 1 creating+peering, 96 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:05 smithi006 bash[26022]: audit 2023-11-17T05:41:04.701214+0000 mon.smithi006 (mon.0) 636 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:05 smithi006 bash[26022]: audit 2023-11-17T05:41:04.835071+0000 mon.smithi006 (mon.0) 637 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:07 smithi006 bash[26022]: cluster 2023-11-17T05:41:06.602025+0000 mgr.smithi006.odexej (mgr.14176) 295 : cluster [DBG] pgmap v259: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.053481+0000 mon.smithi006 (mon.0) 638 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.055524+0000 mon.smithi006 (mon.0) 639 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:22.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.056918+0000 mon.smithi006 (mon.0) 640 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:22.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.065311+0000 mon.smithi006 (mon.0) 641 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.075905+0000 mon.smithi006 (mon.0) 642 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:22.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.093934+0000 mon.smithi006 (mon.0) 643 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.095421+0000 mon.smithi006 (mon.0) 644 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi156.gmiiei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.101572+0000 mon.smithi006 (mon.0) 645 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi156.gmiiei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: audit 2023-11-17T05:41:08.102819+0000 mon.smithi006 (mon.0) 646 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: cephadm 2023-11-17T05:41:08.104055+0000 mgr.smithi006.odexej (mgr.14176) 296 : cephadm [INF] Deploying daemon mds.foofs.smithi156.gmiiei on smithi156 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:09 smithi006 bash[26022]: cluster 2023-11-17T05:41:08.602965+0000 mgr.smithi006.odexej (mgr.14176) 297 : cluster [DBG] pgmap v260: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: audit 2023-11-17T05:41:10.377132+0000 mon.smithi006 (mon.0) 647 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: audit 2023-11-17T05:41:10.379890+0000 mon.smithi006 (mon.0) 648 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi006.rwbcnj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: audit 2023-11-17T05:41:10.386849+0000 mon.smithi006 (mon.0) 649 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi006.rwbcnj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: audit 2023-11-17T05:41:10.388583+0000 mon.smithi006 (mon.0) 650 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: cephadm 2023-11-17T05:41:10.390006+0000 mgr.smithi006.odexej (mgr.14176) 298 : cephadm [INF] Deploying daemon mds.foofs.smithi006.rwbcnj on smithi006 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:11 smithi006 bash[26022]: cluster 2023-11-17T05:41:10.604409+0000 mgr.smithi006.odexej (mgr.14176) 299 : cluster [DBG] pgmap v261: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.390734+0000 mon.smithi006 (mon.0) 651 : cluster [DBG] mds.? [v2:172.21.15.156:6832/845746766,v1:172.21.15.156:6833/845746766] up:boot 2023-11-17T05:42:22.858 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.390868+0000 mon.smithi006 (mon.0) 652 : cluster [INF] daemon mds.foofs.smithi156.gmiiei assigned to filesystem foofs as rank 0 (now has 1 ranks) 2023-11-17T05:42:22.859 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.391056+0000 mon.smithi006 (mon.0) 653 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T05:42:22.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.391109+0000 mon.smithi006 (mon.0) 654 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-17T05:42:22.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.391141+0000 mon.smithi006 (mon.0) 655 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:22.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.391549+0000 mon.smithi006 (mon.0) 656 : cluster [DBG] fsmap foofs:0 1 up:standby 2023-11-17T05:42:22.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.391732+0000 mon.smithi006 (mon.0) 657 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi156.gmiiei"}]: dispatch 2023-11-17T05:42:22.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.397563+0000 mon.smithi006 (mon.0) 658 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:creating} 2023-11-17T05:42:22.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.415051+0000 mon.smithi006 (mon.0) 659 : cluster [INF] daemon mds.foofs.smithi156.gmiiei is now active in filesystem foofs as rank 0 2023-11-17T05:42:22.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.596662+0000 mon.smithi006 (mon.0) 660 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:22.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cluster 2023-11-17T05:41:12.605140+0000 mgr.smithi006.odexej (mgr.14176) 300 : cluster [DBG] pgmap v262: 97 pgs: 97 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.605257+0000 mon.smithi006 (mon.0) 661 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cephadm 2023-11-17T05:41:12.635652+0000 mgr.smithi006.odexej (mgr.14176) 301 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi006.uasswy 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.636106+0000 mon.smithi006 (mon.0) 662 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.642261+0000 mon.smithi006 (mon.0) 663 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cephadm 2023-11-17T05:41:12.643025+0000 mgr.smithi006.odexej (mgr.14176) 302 : cephadm [INF] Ensuring nfs.foo.0 is in the ganesha grace table 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.643350+0000 mon.smithi006 (mon.0) 664 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.649091+0000 mon.smithi006 (mon.0) 665 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T05:42:23.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.650591+0000 mon.smithi006 (mon.0) 666 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.706051+0000 mon.smithi006 (mon.0) 667 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.710097+0000 mon.smithi006 (mon.0) 668 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cephadm 2023-11-17T05:41:12.775035+0000 mgr.smithi006.odexej (mgr.14176) 303 : cephadm [INF] Rados config object exists: conf-nfs.foo 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cephadm 2023-11-17T05:41:12.775308+0000 mgr.smithi006.odexej (mgr.14176) 304 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi006.uasswy-rgw 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.776111+0000 mon.smithi006 (mon.0) 669 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.784839+0000 mon.smithi006 (mon.0) 670 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi006.uasswy-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: audit 2023-11-17T05:41:12.795724+0000 mon.smithi006 (mon.0) 671 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:13 smithi006 bash[26022]: cephadm 2023-11-17T05:41:12.797779+0000 mgr.smithi006.odexej (mgr.14176) 305 : cephadm [INF] Deploying daemon nfs.foo.0.0.smithi006.uasswy on smithi006 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:14 smithi006 bash[26022]: cluster 2023-11-17T05:41:13.402408+0000 mon.smithi006 (mon.0) 672 : cluster [DBG] mds.? [v2:172.21.15.156:6832/845746766,v1:172.21.15.156:6833/845746766] up:active 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:14 smithi006 bash[26022]: cluster 2023-11-17T05:41:13.402512+0000 mon.smithi006 (mon.0) 673 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: cluster 2023-11-17T05:41:14.606578+0000 mgr.smithi006.odexej (mgr.14176) 306 : cluster [DBG] pgmap v263: 97 pgs: 97 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1.2 KiB/s wr, 3 op/s 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: cluster 2023-11-17T05:41:14.613621+0000 mon.smithi006 (mon.0) 674 : cluster [DBG] mds.? [v2:172.21.15.6:6834/3961950015,v1:172.21.15.6:6835/3961950015] up:boot 2023-11-17T05:42:23.012 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: cluster 2023-11-17T05:41:14.614394+0000 mon.smithi006 (mon.0) 675 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 1 up:standby 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: audit 2023-11-17T05:41:14.614639+0000 mon.smithi006 (mon.0) 676 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi006.rwbcnj"}]: dispatch 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: cluster 2023-11-17T05:41:14.621360+0000 mon.smithi006 (mon.0) 677 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi156.gmiiei=up:active} 1 up:standby 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: audit 2023-11-17T05:41:15.028432+0000 mon.smithi006 (mon.0) 678 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: audit 2023-11-17T05:41:15.254372+0000 mon.smithi006 (mon.0) 679 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:15 smithi006 bash[26022]: audit 2023-11-17T05:41:15.264265+0000 mon.smithi006 (mon.0) 680 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:16 smithi006 bash[26022]: cephadm 2023-11-17T05:41:15.282215+0000 mgr.smithi006.odexej (mgr.14176) 307 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.uylhhq on smithi156 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:17 smithi006 bash[26022]: cluster 2023-11-17T05:41:16.607440+0000 mgr.smithi006.odexej (mgr.14176) 308 : cluster [DBG] pgmap v264: 97 pgs: 97 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 1023 B/s wr, 3 op/s 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:19 smithi006 bash[26022]: cluster 2023-11-17T05:41:18.608420+0000 mgr.smithi006.odexej (mgr.14176) 309 : cluster [DBG] pgmap v265: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:19 smithi006 bash[26022]: audit 2023-11-17T05:41:18.989070+0000 mon.smithi006 (mon.0) 681 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.uylhhq"}]: dispatch 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: cephadm 2023-11-17T05:41:18.987793+0000 mgr.smithi006.odexej (mgr.14176) 310 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: cephadm 2023-11-17T05:41:18.988395+0000 mgr.smithi006.odexej (mgr.14176) 311 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: cephadm 2023-11-17T05:41:18.990904+0000 mgr.smithi006.odexej (mgr.14176) 312 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.uylhhq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-uylhhq 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.uylhhq 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.uylhhq ... 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: cephadm 2023-11-17T05:41:18.996324+0000 mgr.smithi006.odexej (mgr.14176) 313 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.iyogat on smithi006 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: audit 2023-11-17T05:41:19.701781+0000 mon.smithi006 (mon.0) 682 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:20 smithi006 bash[26022]: audit 2023-11-17T05:41:20.039796+0000 mon.smithi006 (mon.0) 683 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:21 smithi006 bash[26022]: cluster 2023-11-17T05:41:20.609550+0000 mgr.smithi006.odexej (mgr.14176) 314 : cluster [DBG] pgmap v266: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: cluster 2023-11-17T05:41:22.610350+0000 mgr.smithi006.odexej (mgr.14176) 315 : cluster [DBG] pgmap v267: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 1.7 KiB/s wr, 5 op/s 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: cephadm 2023-11-17T05:41:22.945695+0000 mgr.smithi006.odexej (mgr.14176) 316 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:23.709 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.710 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.710 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:23.710 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.710 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: cephadm 2023-11-17T05:41:22.946273+0000 mgr.smithi006.odexej (mgr.14176) 317 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: audit 2023-11-17T05:41:22.947034+0000 mon.smithi006 (mon.0) 684 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.iyogat"}]: dispatch 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: audit 2023-11-17T05:41:22.958090+0000 mon.smithi006 (mon.0) 685 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:23 smithi006 bash[26022]: audit 2023-11-17T05:41:22.962407+0000 mon.smithi006 (mon.0) 686 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: cephadm 2023-11-17T05:41:22.948812+0000 mgr.smithi006.odexej (mgr.14176) 318 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.iyogat on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iyogat 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iyogat 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iyogat ... 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: cephadm 2023-11-17T05:41:22.958579+0000 mgr.smithi006.odexej (mgr.14176) 319 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.wxztob on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: cephadm 2023-11-17T05:41:22.961034+0000 mgr.smithi006.odexej (mgr.14176) 320 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ticlht on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: cluster 2023-11-17T05:41:22.962109+0000 mgr.smithi006.odexej (mgr.14176) 321 : cluster [DBG] pgmap v268: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:24 smithi006 bash[26022]: cluster 2023-11-17T05:41:23.955933+0000 mon.smithi006 (mon.0) 687 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:26 smithi006 bash[26022]: cluster 2023-11-17T05:41:24.963457+0000 mgr.smithi006.odexej (mgr.14176) 322 : cluster [DBG] pgmap v269: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 B/s wr, 2 op/s 2023-11-17T05:42:23.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:26 smithi006 bash[26022]: audit 2023-11-17T05:41:25.052122+0000 mon.smithi006 (mon.0) 688 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:28 smithi006 bash[26022]: cluster 2023-11-17T05:41:26.964437+0000 mgr.smithi006.odexej (mgr.14176) 323 : cluster [DBG] pgmap v270: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 B/s wr, 2 op/s 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:29 smithi006 bash[26022]: audit 2023-11-17T05:41:28.645911+0000 mon.smithi006 (mon.0) 689 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:29 smithi006 bash[26022]: audit 2023-11-17T05:41:29.488253+0000 mon.smithi006 (mon.0) 690 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: cluster 2023-11-17T05:41:28.965535+0000 mgr.smithi006.odexej (mgr.14176) 324 : cluster [DBG] pgmap v271: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 B/s wr, 0 op/s 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: audit 2023-11-17T05:41:29.786531+0000 mon.smithi006 (mon.0) 691 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: audit 2023-11-17T05:41:29.788099+0000 mon.smithi006 (mon.0) 692 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: audit 2023-11-17T05:41:29.797618+0000 mon.smithi006 (mon.0) 693 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: cluster 2023-11-17T05:41:29.798817+0000 mgr.smithi006.odexej (mgr.14176) 325 : cluster [DBG] pgmap v272: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 111 B/s rd, 0 op/s 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: audit 2023-11-17T05:41:29.804010+0000 mon.smithi006 (mon.0) 694 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: audit 2023-11-17T05:41:29.820535+0000 mon.smithi006 (mon.0) 695 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:30 smithi006 bash[26022]: cephadm 2023-11-17T05:41:29.825359+0000 mgr.smithi006.odexej (mgr.14176) 326 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dnwbpo on smithi156 2023-11-17T05:42:23.724 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:31 smithi006 bash[26022]: cluster 2023-11-17T05:41:30.794210+0000 mon.smithi006 (mon.0) 696 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:31 smithi006 bash[26022]: cluster 2023-11-17T05:41:30.794276+0000 mon.smithi006 (mon.0) 697 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:32 smithi006 bash[26022]: cluster 2023-11-17T05:41:31.799598+0000 mgr.smithi006.odexej (mgr.14176) 327 : cluster [DBG] pgmap v273: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:33 smithi006 bash[26022]: audit 2023-11-17T05:41:33.536046+0000 mon.smithi006 (mon.0) 698 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dnwbpo"}]: dispatch 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: cephadm 2023-11-17T05:41:33.534484+0000 mgr.smithi006.odexej (mgr.14176) 328 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.785 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: cephadm 2023-11-17T05:41:33.535089+0000 mgr.smithi006.odexej (mgr.14176) 329 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: cephadm 2023-11-17T05:41:33.537712+0000 mgr.smithi006.odexej (mgr.14176) 330 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dnwbpo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dnwbpo 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dnwbpo 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dnwbpo ... 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: cephadm 2023-11-17T05:41:33.541038+0000 mgr.smithi006.odexej (mgr.14176) 331 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.kyltxj on smithi006 2023-11-17T05:42:23.787 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:34 smithi006 bash[26022]: cluster 2023-11-17T05:41:33.800237+0000 mgr.smithi006.odexej (mgr.14176) 332 : cluster [DBG] pgmap v274: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:36 smithi006 bash[26022]: cluster 2023-11-17T05:41:35.801305+0000 mgr.smithi006.odexej (mgr.14176) 333 : cluster [DBG] pgmap v275: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:37 smithi006 bash[26022]: audit 2023-11-17T05:41:37.268055+0000 mon.smithi006 (mon.0) 699 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.kyltxj"}]: dispatch 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:37 smithi006 bash[26022]: audit 2023-11-17T05:41:37.279759+0000 mon.smithi006 (mon.0) 700 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cephadm 2023-11-17T05:41:37.266820+0000 mgr.smithi006.odexej (mgr.14176) 334 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.811 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cephadm 2023-11-17T05:41:37.267245+0000 mgr.smithi006.odexej (mgr.14176) 335 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cephadm 2023-11-17T05:41:37.269525+0000 mgr.smithi006.odexej (mgr.14176) 336 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.kyltxj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-kyltxj 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.kyltxj 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.kyltxj ... 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.812 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cephadm 2023-11-17T05:41:37.272830+0000 mgr.smithi006.odexej (mgr.14176) 337 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vmmwrr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cephadm 2023-11-17T05:41:37.275352+0000 mgr.smithi006.odexej (mgr.14176) 338 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cluster 2023-11-17T05:41:37.276759+0000 mgr.smithi006.odexej (mgr.14176) 339 : cluster [DBG] pgmap v276: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:38 smithi006 bash[26022]: cluster 2023-11-17T05:41:37.858504+0000 mon.smithi006 (mon.0) 701 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:40 smithi006 bash[26022]: cluster 2023-11-17T05:41:39.278117+0000 mgr.smithi006.odexej (mgr.14176) 340 : cluster [DBG] pgmap v277: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:40 smithi006 bash[26022]: audit 2023-11-17T05:41:40.072676+0000 mon.smithi006 (mon.0) 702 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:42 smithi006 bash[26022]: cluster 2023-11-17T05:41:41.279257+0000 mgr.smithi006.odexej (mgr.14176) 341 : cluster [DBG] pgmap v278: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:42.984191+0000 mon.smithi006 (mon.0) 703 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: cluster 2023-11-17T05:41:43.280383+0000 mgr.smithi006.odexej (mgr.14176) 342 : cluster [DBG] pgmap v279: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T05:42:23.813 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.770942+0000 mon.smithi006 (mon.0) 704 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.843 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.772729+0000 mon.smithi006 (mon.0) 705 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:23.843 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.774526+0000 mon.smithi006 (mon.0) 706 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:23.843 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.784531+0000 mon.smithi006 (mon.0) 707 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.843 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: cluster 2023-11-17T05:41:43.785722+0000 mgr.smithi006.odexej (mgr.14176) 343 : cluster [DBG] pgmap v280: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:42:23.843 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.793473+0000 mon.smithi006 (mon.0) 708 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: audit 2023-11-17T05:41:43.813746+0000 mon.smithi006 (mon.0) 709 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:43 smithi006 bash[26022]: cephadm 2023-11-17T05:41:43.818239+0000 mgr.smithi006.odexej (mgr.14176) 344 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xaaadw on smithi156 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:44 smithi006 bash[26022]: cluster 2023-11-17T05:41:43.980810+0000 mon.smithi006 (mon.0) 710 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:44 smithi006 bash[26022]: cluster 2023-11-17T05:41:43.980897+0000 mon.smithi006 (mon.0) 711 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:46 smithi006 bash[26022]: cluster 2023-11-17T05:41:45.786785+0000 mgr.smithi006.odexej (mgr.14176) 345 : cluster [DBG] pgmap v281: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:42:23.844 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:47 smithi006 bash[26022]: audit 2023-11-17T05:41:47.550718+0000 mon.smithi006 (mon.0) 712 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xaaadw"}]: dispatch 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: cephadm 2023-11-17T05:41:47.549525+0000 mgr.smithi006.odexej (mgr.14176) 346 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.865 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: cephadm 2023-11-17T05:41:47.550063+0000 mgr.smithi006.odexej (mgr.14176) 347 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: cephadm 2023-11-17T05:41:47.552522+0000 mgr.smithi006.odexej (mgr.14176) 348 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw 2023-11-17T05:42:23.867 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xaaadw ... 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: cephadm 2023-11-17T05:41:47.557721+0000 mgr.smithi006.odexej (mgr.14176) 349 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.bnvxvk on smithi006 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:48 smithi006 bash[26022]: cluster 2023-11-17T05:41:47.787381+0000 mgr.smithi006.odexej (mgr.14176) 350 : cluster [DBG] pgmap v282: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:50 smithi006 bash[26022]: cluster 2023-11-17T05:41:49.788478+0000 mgr.smithi006.odexej (mgr.14176) 351 : cluster [DBG] pgmap v283: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cephadm 2023-11-17T05:41:51.306868+0000 mgr.smithi006.odexej (mgr.14176) 352 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.868 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:23.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cephadm 2023-11-17T05:41:51.307438+0000 mgr.smithi006.odexej (mgr.14176) 353 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: audit 2023-11-17T05:41:51.308174+0000 mon.smithi006 (mon.0) 713 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.bnvxvk"}]: dispatch 2023-11-17T05:42:23.877 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cephadm 2023-11-17T05:41:51.309934+0000 mgr.smithi006.odexej (mgr.14176) 354 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.877 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.bnvxvk ... 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:23.878 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cephadm 2023-11-17T05:41:51.314376+0000 mgr.smithi006.odexej (mgr.14176) 355 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cephadm 2023-11-17T05:41:51.319179+0000 mgr.smithi006.odexej (mgr.14176) 356 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:52 smithi006 bash[26022]: cluster 2023-11-17T05:41:51.320369+0000 mgr.smithi006.odexej (mgr.14176) 357 : cluster [DBG] pgmap v284: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:53 smithi006 bash[26022]: cluster 2023-11-17T05:41:52.259720+0000 mon.smithi006 (mon.0) 714 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:54 smithi006 bash[26022]: cluster 2023-11-17T05:41:53.321526+0000 mgr.smithi006.odexej (mgr.14176) 358 : cluster [DBG] pgmap v285: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:56 smithi006 bash[26022]: audit 2023-11-17T05:41:55.092689+0000 mon.smithi006 (mon.0) 715 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:56 smithi006 bash[26022]: cluster 2023-11-17T05:41:55.322845+0000 mgr.smithi006.odexej (mgr.14176) 359 : cluster [DBG] pgmap v286: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:41:58 smithi006 bash[26022]: cluster 2023-11-17T05:41:57.324004+0000 mgr.smithi006.odexej (mgr.14176) 360 : cluster [DBG] pgmap v287: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:00 smithi006 bash[26022]: cluster 2023-11-17T05:41:59.325412+0000 mgr.smithi006.odexej (mgr.14176) 361 : cluster [DBG] pgmap v288: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:02 smithi006 bash[26022]: cluster 2023-11-17T05:42:01.326453+0000 mgr.smithi006.odexej (mgr.14176) 362 : cluster [DBG] pgmap v289: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:04 smithi006 bash[26022]: cluster 2023-11-17T05:42:03.327603+0000 mgr.smithi006.odexej (mgr.14176) 363 : cluster [DBG] pgmap v290: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:05 smithi006 bash[26022]: audit 2023-11-17T05:42:04.703247+0000 mon.smithi006 (mon.0) 716 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:23.879 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:06 smithi006 bash[26022]: cluster 2023-11-17T05:42:05.328927+0000 mgr.smithi006.odexej (mgr.14176) 364 : cluster [DBG] pgmap v291: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:08 smithi006 bash[26022]: cluster 2023-11-17T05:42:07.330048+0000 mgr.smithi006.odexej (mgr.14176) 365 : cluster [DBG] pgmap v292: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:10 smithi006 bash[26022]: cluster 2023-11-17T05:42:09.331492+0000 mgr.smithi006.odexej (mgr.14176) 366 : cluster [DBG] pgmap v293: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:12 smithi006 bash[26022]: cluster 2023-11-17T05:42:11.332521+0000 mgr.smithi006.odexej (mgr.14176) 367 : cluster [DBG] pgmap v294: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:14 smithi006 bash[26022]: cluster 2023-11-17T05:42:13.333609+0000 mgr.smithi006.odexej (mgr.14176) 368 : cluster [DBG] pgmap v295: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:16 smithi006 bash[26022]: cluster 2023-11-17T05:42:15.334942+0000 mgr.smithi006.odexej (mgr.14176) 369 : cluster [DBG] pgmap v296: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:18 smithi006 bash[26022]: cluster 2023-11-17T05:42:17.336008+0000 mgr.smithi006.odexej (mgr.14176) 370 : cluster [DBG] pgmap v297: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:20 smithi006 bash[26022]: cluster 2023-11-17T05:42:19.337439+0000 mgr.smithi006.odexej (mgr.14176) 371 : cluster [DBG] pgmap v298: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:23.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:22 smithi006 bash[26022]: cluster 2023-11-17T05:42:21.338458+0000 mgr.smithi006.odexej (mgr.14176) 372 : cluster [DBG] pgmap v299: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:24.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:24 smithi006 bash[26022]: cluster 2023-11-17T05:42:23.339631+0000 mgr.smithi006.odexej (mgr.14176) 373 : cluster [DBG] pgmap v300: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:24 smithi156 bash[29655]: cluster 2023-11-17T05:42:23.339631+0000 mgr.smithi006.odexej (mgr.14176) 373 : cluster [DBG] pgmap v300: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:25.982 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-11-17T05:42:25.982 INFO:teuthology.orchestra.run.smithi006.stdout: "bind": "/fake", 2023-11-17T05:42:25.983 INFO:teuthology.orchestra.run.smithi006.stdout: "fs": "foofs", 2023-11-17T05:42:25.983 INFO:teuthology.orchestra.run.smithi006.stdout: "path": "/", 2023-11-17T05:42:25.983 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster": "foo", 2023-11-17T05:42:25.983 INFO:teuthology.orchestra.run.smithi006.stdout: "mode": "RW" 2023-11-17T05:42:25.983 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-11-17T05:42:26.617 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2023-11-17T05:42:26.628 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2023-11-17T05:42:26.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:26.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:26 smithi156 bash[29655]: cluster 2023-11-17T05:42:25.340926+0000 mgr.smithi006.odexej (mgr.14176) 374 : cluster [DBG] pgmap v301: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:26.722 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:26 smithi156 bash[29655]: audit 2023-11-17T05:42:25.893355+0000 mgr.smithi006.odexej (mgr.14176) 375 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:42:26.722 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:26 smithi156 bash[29655]: audit 2023-11-17T05:42:25.963144+0000 mon.smithi006 (mon.0) 717 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2023-11-17T05:42:26.722 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:26 smithi156 bash[29655]: audit 2023-11-17T05:42:25.969427+0000 mon.smithi006 (mon.0) 718 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2023-11-17T05:42:26.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:26 smithi006 bash[26022]: cluster 2023-11-17T05:42:25.340926+0000 mgr.smithi006.odexej (mgr.14176) 374 : cluster [DBG] pgmap v301: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:26.722 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:26 smithi006 bash[26022]: audit 2023-11-17T05:42:25.893355+0000 mgr.smithi006.odexej (mgr.14176) 375 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:42:26.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:26 smithi006 bash[26022]: audit 2023-11-17T05:42:25.963144+0000 mon.smithi006 (mon.0) 717 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2023-11-17T05:42:26.723 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:26 smithi006 bash[26022]: audit 2023-11-17T05:42:25.969427+0000 mon.smithi006 (mon.0) 718 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2023-11-17T05:42:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:28 smithi156 bash[29655]: cluster 2023-11-17T05:42:27.341979+0000 mgr.smithi006.odexej (mgr.14176) 376 : cluster [DBG] pgmap v302: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:28.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:28 smithi006 bash[26022]: cluster 2023-11-17T05:42:27.341979+0000 mgr.smithi006.odexej (mgr.14176) 376 : cluster [DBG] pgmap v302: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:29.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:29.901 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:29.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:29 smithi006 bash[26022]: cluster 2023-11-17T05:42:28.416729+0000 mon.smithi006 (mon.0) 719 : cluster [DBG] mgrmap e20: smithi006.odexej(active, since 8m), standbys: smithi156.fphsnp 2023-11-17T05:42:29.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:29 smithi156 bash[29655]: cluster 2023-11-17T05:42:28.416729+0000 mon.smithi006 (mon.0) 719 : cluster [DBG] mgrmap e20: smithi006.odexej(active, since 8m), standbys: smithi156.fphsnp 2023-11-17T05:42:30.543 INFO:tasks.cephadm:nfs.foo has 1/1 2023-11-17T05:42:30.543 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2023-11-17T05:42:30.553 INFO:tasks.cephadm:Waiting for ceph service ingress.nfs.foo to start (timeout 300)... 2023-11-17T05:42:30.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:30.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:30 smithi006 bash[26022]: cluster 2023-11-17T05:42:29.343337+0000 mgr.smithi006.odexej (mgr.14176) 377 : cluster [DBG] pgmap v303: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:30.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:30 smithi006 bash[26022]: audit 2023-11-17T05:42:29.889022+0000 mgr.smithi006.odexej (mgr.14176) 378 : audit [DBG] from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:30.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:30 smithi156 bash[29655]: cluster 2023-11-17T05:42:29.343337+0000 mgr.smithi006.odexej (mgr.14176) 377 : cluster [DBG] pgmap v303: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:30.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:30 smithi156 bash[29655]: audit 2023-11-17T05:42:29.889022+0000 mgr.smithi006.odexej (mgr.14176) 378 : audit [DBG] from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:32 smithi006 bash[26022]: cluster 2023-11-17T05:42:31.344424+0000 mgr.smithi006.odexej (mgr.14176) 379 : cluster [DBG] pgmap v304: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:32.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:32 smithi156 bash[29655]: cluster 2023-11-17T05:42:31.344424+0000 mgr.smithi006.odexej (mgr.14176) 379 : cluster [DBG] pgmap v304: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:33.720 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:33.720 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:34.342 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:34.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:34 smithi006 bash[26022]: cluster 2023-11-17T05:42:33.345396+0000 mgr.smithi006.odexej (mgr.14176) 380 : cluster [DBG] pgmap v305: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 596 B/s wr, 1 op/s 2023-11-17T05:42:34.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:34 smithi006 bash[26022]: audit 2023-11-17T05:42:33.702323+0000 mgr.smithi006.odexej (mgr.14176) 381 : audit [DBG] from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:34.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:34 smithi156 bash[29655]: cluster 2023-11-17T05:42:33.345396+0000 mgr.smithi006.odexej (mgr.14176) 380 : cluster [DBG] pgmap v305: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 596 B/s wr, 1 op/s 2023-11-17T05:42:34.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:34 smithi156 bash[29655]: audit 2023-11-17T05:42:33.702323+0000 mgr.smithi006.odexej (mgr.14176) 381 : audit [DBG] from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:35.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:36.633 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:36 smithi006 bash[26022]: cluster 2023-11-17T05:42:35.346814+0000 mgr.smithi006.odexej (mgr.14176) 382 : cluster [DBG] pgmap v306: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:36.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:36 smithi156 bash[29655]: cluster 2023-11-17T05:42:35.346814+0000 mgr.smithi006.odexej (mgr.14176) 382 : cluster [DBG] pgmap v306: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:38.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:38.382 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:38.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:38 smithi156 bash[29655]: cluster 2023-11-17T05:42:37.347931+0000 mgr.smithi006.odexej (mgr.14176) 383 : cluster [DBG] pgmap v307: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:38.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:38 smithi006 bash[26022]: cluster 2023-11-17T05:42:37.347931+0000 mgr.smithi006.odexej (mgr.14176) 383 : cluster [DBG] pgmap v307: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:38.973 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:39.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:39 smithi156 bash[29655]: audit 2023-11-17T05:42:38.371173+0000 mgr.smithi006.odexej (mgr.14176) 384 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:39.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:39 smithi006 bash[26022]: audit 2023-11-17T05:42:38.371173+0000 mgr.smithi006.odexej (mgr.14176) 384 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:39.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:40 smithi156 bash[29655]: cluster 2023-11-17T05:42:39.349311+0000 mgr.smithi006.odexej (mgr.14176) 385 : cluster [DBG] pgmap v308: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:40.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:40 smithi006 bash[26022]: cluster 2023-11-17T05:42:39.349311+0000 mgr.smithi006.odexej (mgr.14176) 385 : cluster [DBG] pgmap v308: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 596 B/s wr, 0 op/s 2023-11-17T05:42:42.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:42 smithi006 bash[26022]: cluster 2023-11-17T05:42:41.350718+0000 mgr.smithi006.odexej (mgr.14176) 386 : cluster [DBG] pgmap v309: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:42.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:42 smithi156 bash[29655]: cluster 2023-11-17T05:42:41.350718+0000 mgr.smithi006.odexej (mgr.14176) 386 : cluster [DBG] pgmap v309: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:43.005 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:43.005 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:43.618 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:44.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:44.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:44 smithi006 bash[26022]: audit 2023-11-17T05:42:42.989736+0000 mgr.smithi006.odexej (mgr.14176) 387 : audit [DBG] from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:44.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:44 smithi006 bash[26022]: cluster 2023-11-17T05:42:43.351950+0000 mgr.smithi006.odexej (mgr.14176) 388 : cluster [DBG] pgmap v310: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:44 smithi156 bash[29655]: audit 2023-11-17T05:42:42.989736+0000 mgr.smithi006.odexej (mgr.14176) 387 : audit [DBG] from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:44 smithi156 bash[29655]: cluster 2023-11-17T05:42:43.351950+0000 mgr.smithi006.odexej (mgr.14176) 388 : cluster [DBG] pgmap v310: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:42:46.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:46 smithi156 bash[29655]: cluster 2023-11-17T05:42:45.353228+0000 mgr.smithi006.odexej (mgr.14176) 389 : cluster [DBG] pgmap v311: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:46.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:46 smithi006 bash[26022]: cluster 2023-11-17T05:42:45.353228+0000 mgr.smithi006.odexej (mgr.14176) 389 : cluster [DBG] pgmap v311: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:47.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:47.629 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:48.182 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:48.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:48 smithi006 bash[26022]: cluster 2023-11-17T05:42:47.354251+0000 mgr.smithi006.odexej (mgr.14176) 390 : cluster [DBG] pgmap v312: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:48.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:48 smithi006 bash[26022]: audit 2023-11-17T05:42:47.616842+0000 mgr.smithi006.odexej (mgr.14176) 391 : audit [DBG] from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:48.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:48 smithi156 bash[29655]: cluster 2023-11-17T05:42:47.354251+0000 mgr.smithi006.odexej (mgr.14176) 390 : cluster [DBG] pgmap v312: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:48.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:48 smithi156 bash[29655]: audit 2023-11-17T05:42:47.616842+0000 mgr.smithi006.odexej (mgr.14176) 391 : audit [DBG] from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:49.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:50.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:50 smithi006 bash[26022]: cluster 2023-11-17T05:42:49.355582+0000 mgr.smithi006.odexej (mgr.14176) 392 : cluster [DBG] pgmap v313: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:50.932 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:50 smithi156 bash[29655]: cluster 2023-11-17T05:42:49.355582+0000 mgr.smithi006.odexej (mgr.14176) 392 : cluster [DBG] pgmap v313: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:52.269 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:52.270 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:37.275204Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rnnpzi on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.323775+0000 mon.smithi006 (mon.0) 720 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: cluster 2023-11-17T05:42:51.356589+0000 mgr.smithi006.odexej (mgr.14176) 393 : cluster [DBG] pgmap v314: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.626810+0000 mon.smithi006 (mon.0) 721 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.921955+0000 mon.smithi006 (mon.0) 722 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.923646+0000 mon.smithi006 (mon.0) 723 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.931313+0000 mon.smithi006 (mon.0) 724 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: cluster 2023-11-17T05:42:51.932040+0000 mgr.smithi006.odexej (mgr.14176) 394 : cluster [DBG] pgmap v315: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:52.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.937636+0000 mon.smithi006 (mon.0) 725 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:52.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: audit 2023-11-17T05:42:51.951073+0000 mon.smithi006 (mon.0) 726 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:52 smithi006 bash[26022]: cephadm 2023-11-17T05:42:51.954843+0000 mgr.smithi006.odexej (mgr.14176) 395 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qimxpq on smithi156 2023-11-17T05:42:52.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.323775+0000 mon.smithi006 (mon.0) 720 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:42:52.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: cluster 2023-11-17T05:42:51.356589+0000 mgr.smithi006.odexej (mgr.14176) 393 : cluster [DBG] pgmap v314: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:42:52.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.626810+0000 mon.smithi006 (mon.0) 721 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.921955+0000 mon.smithi006 (mon.0) 722 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.923646+0000 mon.smithi006 (mon.0) 723 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.931313+0000 mon.smithi006 (mon.0) 724 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: cluster 2023-11-17T05:42:51.932040+0000 mgr.smithi006.odexej (mgr.14176) 394 : cluster [DBG] pgmap v315: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.937636+0000 mon.smithi006 (mon.0) 725 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: audit 2023-11-17T05:42:51.951073+0000 mon.smithi006 (mon.0) 726 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:42:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:52 smithi156 bash[29655]: cephadm 2023-11-17T05:42:51.954843+0000 mgr.smithi006.odexej (mgr.14176) 395 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qimxpq on smithi156 2023-11-17T05:42:52.900 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:53.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:53 smithi156 bash[29655]: audit 2023-11-17T05:42:52.254214+0000 mgr.smithi006.odexej (mgr.14176) 396 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:53.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:53 smithi156 bash[29655]: cluster 2023-11-17T05:42:52.278876+0000 mon.smithi006 (mon.0) 727 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:53.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:53 smithi156 bash[29655]: cluster 2023-11-17T05:42:52.278945+0000 mon.smithi006 (mon.0) 728 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:53.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:53 smithi006 bash[26022]: audit 2023-11-17T05:42:52.254214+0000 mgr.smithi006.odexej (mgr.14176) 396 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:53.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:53 smithi006 bash[26022]: cluster 2023-11-17T05:42:52.278876+0000 mon.smithi006 (mon.0) 727 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:42:53.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:53 smithi006 bash[26022]: cluster 2023-11-17T05:42:52.278945+0000 mon.smithi006 (mon.0) 728 : cluster [INF] Cluster is now healthy 2023-11-17T05:42:53.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:42:54.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:54 smithi006 bash[26022]: cluster 2023-11-17T05:42:53.932889+0000 mgr.smithi006.odexej (mgr.14176) 397 : cluster [DBG] pgmap v316: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:54.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:54 smithi156 bash[29655]: cluster 2023-11-17T05:42:53.932889+0000 mgr.smithi006.odexej (mgr.14176) 397 : cluster [DBG] pgmap v316: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:55.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:55 smithi156 bash[29655]: audit 2023-11-17T05:42:55.687389+0000 mon.smithi006 (mon.0) 729 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qimxpq"}]: dispatch 2023-11-17T05:42:56.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:55 smithi006 bash[26022]: audit 2023-11-17T05:42:55.687389+0000 mon.smithi006 (mon.0) 729 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qimxpq"}]: dispatch 2023-11-17T05:42:57.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: cephadm 2023-11-17T05:42:55.685944+0000 mgr.smithi006.odexej (mgr.14176) 398 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:57.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:42:57.245 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:57.245 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:42:57.245 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.245 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.245 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.246 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: cephadm 2023-11-17T05:42:55.686513+0000 mgr.smithi006.odexej (mgr.14176) 399 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: cephadm 2023-11-17T05:42:55.689363+0000 mgr.smithi006.odexej (mgr.14176) 400 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: cephadm 2023-11-17T05:42:55.693900+0000 mgr.smithi006.odexej (mgr.14176) 401 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.attjol on smithi006 2023-11-17T05:42:57.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:56 smithi006 bash[26022]: cluster 2023-11-17T05:42:55.933985+0000 mgr.smithi006.odexej (mgr.14176) 402 : cluster [DBG] pgmap v317: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:57.252 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:42:57.252 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:47.552292Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xaaadw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xaaadw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xaaadw\nDeploy daemon haproxy.nfs.foo.smithi156.xaaadw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.309623Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.bnvxvk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-bnvxvk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.bnvxvk\nDeploy daemon haproxy.nfs.foo.smithi006.bnvxvk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:41:51.314145Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nhbxut on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:42:57.252 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: cephadm 2023-11-17T05:42:55.685944+0000 mgr.smithi006.odexej (mgr.14176) 398 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.252 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.253 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.254 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.254 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.254 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.255 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.259 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.259 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.259 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.259 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.259 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: cephadm 2023-11-17T05:42:55.686513+0000 mgr.smithi006.odexej (mgr.14176) 399 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: cephadm 2023-11-17T05:42:55.689363+0000 mgr.smithi006.odexej (mgr.14176) 400 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq 2023-11-17T05:42:57.260 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.261 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:42:57.261 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq 2023-11-17T05:42:57.261 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qimxpq ... 2023-11-17T05:42:57.262 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.262 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:42:57.262 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:42:57.262 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:42:57.264 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:42:57.264 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: cephadm 2023-11-17T05:42:55.693900+0000 mgr.smithi006.odexej (mgr.14176) 401 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.attjol on smithi006 2023-11-17T05:42:57.264 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:56 smithi156 bash[29655]: cluster 2023-11-17T05:42:55.933985+0000 mgr.smithi006.odexej (mgr.14176) 402 : cluster [DBG] pgmap v317: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:57.762 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:42:58.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:58 smithi006 bash[26022]: audit 2023-11-17T05:42:57.170839+0000 mgr.smithi006.odexej (mgr.14176) 403 : audit [DBG] from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:58.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:58 smithi006 bash[26022]: cluster 2023-11-17T05:42:57.934840+0000 mgr.smithi006.odexej (mgr.14176) 404 : cluster [DBG] pgmap v318: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:58.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:58 smithi156 bash[29655]: audit 2023-11-17T05:42:57.170839+0000 mgr.smithi006.odexej (mgr.14176) 403 : audit [DBG] from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:42:58.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:58 smithi156 bash[29655]: cluster 2023-11-17T05:42:57.934840+0000 mgr.smithi006.odexej (mgr.14176) 404 : cluster [DBG] pgmap v318: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:42:58.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:00.028 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:42:59 smithi006 bash[26022]: audit 2023-11-17T05:42:59.794955+0000 mon.smithi006 (mon.0) 730 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.attjol"}]: dispatch 2023-11-17T05:43:00.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:42:59 smithi156 bash[29655]: audit 2023-11-17T05:42:59.794955+0000 mon.smithi006 (mon.0) 730 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.attjol"}]: dispatch 2023-11-17T05:43:01.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cephadm 2023-11-17T05:42:59.793639+0000 mgr.smithi006.odexej (mgr.14176) 405 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:43:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cephadm 2023-11-17T05:42:59.794214+0000 mgr.smithi006.odexej (mgr.14176) 406 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cephadm 2023-11-17T05:42:59.796475+0000 mgr.smithi006.odexej (mgr.14176) 407 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cephadm 2023-11-17T05:42:59.800326+0000 mgr.smithi006.odexej (mgr.14176) 408 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cephadm 2023-11-17T05:42:59.803769+0000 mgr.smithi006.odexej (mgr.14176) 409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cluster 2023-11-17T05:42:59.804994+0000 mgr.smithi006.odexej (mgr.14176) 410 : cluster [DBG] pgmap v319: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:43:01.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: cluster 2023-11-17T05:42:59.856087+0000 mon.smithi006 (mon.0) 731 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:43:01.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:00 smithi006 bash[26022]: audit 2023-11-17T05:43:00.140750+0000 mon.smithi006 (mon.0) 732 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cephadm 2023-11-17T05:42:59.793639+0000 mgr.smithi006.odexej (mgr.14176) 405 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:43:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cephadm 2023-11-17T05:42:59.794214+0000 mgr.smithi006.odexej (mgr.14176) 406 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cephadm 2023-11-17T05:42:59.796475+0000 mgr.smithi006.odexej (mgr.14176) 407 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.attjol ... 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cephadm 2023-11-17T05:42:59.800326+0000 mgr.smithi006.odexej (mgr.14176) 408 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cephadm 2023-11-17T05:42:59.803769+0000 mgr.smithi006.odexej (mgr.14176) 409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cluster 2023-11-17T05:42:59.804994+0000 mgr.smithi006.odexej (mgr.14176) 410 : cluster [DBG] pgmap v319: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: cluster 2023-11-17T05:42:59.856087+0000 mon.smithi006 (mon.0) 731 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:43:01.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:00 smithi156 bash[29655]: audit 2023-11-17T05:43:00.140750+0000 mon.smithi006 (mon.0) 732 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:43:02.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:02.145 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:02.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:02 smithi006 bash[26022]: cluster 2023-11-17T05:43:01.805805+0000 mgr.smithi006.odexej (mgr.14176) 411 : cluster [DBG] pgmap v320: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:43:02.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:02 smithi156 bash[29655]: cluster 2023-11-17T05:43:01.805805+0000 mgr.smithi006.odexej (mgr.14176) 411 : cluster [DBG] pgmap v320: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:43:02.738 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:03.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:03 smithi156 bash[29655]: audit 2023-11-17T05:43:02.129489+0000 mgr.smithi006.odexej (mgr.14176) 412 : audit [DBG] from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:03.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:03 smithi006 bash[26022]: audit 2023-11-17T05:43:02.129489+0000 mgr.smithi006.odexej (mgr.14176) 412 : audit [DBG] from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:03.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:04 smithi006 bash[26022]: cluster 2023-11-17T05:43:03.806979+0000 mgr.smithi006.odexej (mgr.14176) 413 : cluster [DBG] pgmap v321: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T05:43:04.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:04 smithi156 bash[29655]: cluster 2023-11-17T05:43:03.806979+0000 mgr.smithi006.odexej (mgr.14176) 413 : cluster [DBG] pgmap v321: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T05:43:06.871 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:06.874 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:07.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:06 smithi006 bash[26022]: cluster 2023-11-17T05:43:05.808320+0000 mgr.smithi006.odexej (mgr.14176) 414 : cluster [DBG] pgmap v322: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:07.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:06 smithi156 bash[29655]: cluster 2023-11-17T05:43:05.808320+0000 mgr.smithi006.odexej (mgr.14176) 414 : cluster [DBG] pgmap v322: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:07.439 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:08.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:07 smithi006 bash[26022]: audit 2023-11-17T05:43:06.850803+0000 mgr.smithi006.odexej (mgr.14176) 415 : audit [DBG] from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:08.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:07 smithi156 bash[29655]: audit 2023-11-17T05:43:06.850803+0000 mgr.smithi006.odexej (mgr.14176) 415 : audit [DBG] from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:08.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:09.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:08 smithi006 bash[26022]: cluster 2023-11-17T05:43:07.809482+0000 mgr.smithi006.odexej (mgr.14176) 416 : cluster [DBG] pgmap v323: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:09.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:08 smithi156 bash[29655]: cluster 2023-11-17T05:43:07.809482+0000 mgr.smithi006.odexej (mgr.14176) 416 : cluster [DBG] pgmap v323: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:11.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:10 smithi006 bash[26022]: cluster 2023-11-17T05:43:09.811037+0000 mgr.smithi006.odexej (mgr.14176) 417 : cluster [DBG] pgmap v324: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:11.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:10 smithi156 bash[29655]: cluster 2023-11-17T05:43:09.811037+0000 mgr.smithi006.odexej (mgr.14176) 417 : cluster [DBG] pgmap v324: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:43:11.497 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:11.498 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:12.087 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:12 smithi006 bash[26022]: audit 2023-11-17T05:43:11.487647+0000 mgr.smithi006.odexej (mgr.14176) 418 : audit [DBG] from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:12.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:12 smithi006 bash[26022]: cluster 2023-11-17T05:43:11.812149+0000 mgr.smithi006.odexej (mgr.14176) 419 : cluster [DBG] pgmap v325: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:12.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:12 smithi156 bash[29655]: audit 2023-11-17T05:43:11.487647+0000 mgr.smithi006.odexej (mgr.14176) 418 : audit [DBG] from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:12.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:12 smithi156 bash[29655]: cluster 2023-11-17T05:43:11.812149+0000 mgr.smithi006.odexej (mgr.14176) 419 : cluster [DBG] pgmap v325: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:13.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:15.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:14 smithi006 bash[26022]: cluster 2023-11-17T05:43:13.813287+0000 mgr.smithi006.odexej (mgr.14176) 420 : cluster [DBG] pgmap v326: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:15.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:14 smithi156 bash[29655]: cluster 2023-11-17T05:43:13.813287+0000 mgr.smithi006.odexej (mgr.14176) 420 : cluster [DBG] pgmap v326: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:16.282 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:16.282 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:16.855 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:17.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:16 smithi006 bash[26022]: cluster 2023-11-17T05:43:15.814636+0000 mgr.smithi006.odexej (mgr.14176) 421 : cluster [DBG] pgmap v327: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:17.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:16 smithi156 bash[29655]: cluster 2023-11-17T05:43:15.814636+0000 mgr.smithi006.odexej (mgr.14176) 421 : cluster [DBG] pgmap v327: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:17.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:18.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:17 smithi006 bash[26022]: audit 2023-11-17T05:43:16.266330+0000 mgr.smithi006.odexej (mgr.14176) 422 : audit [DBG] from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:18.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:17 smithi156 bash[29655]: audit 2023-11-17T05:43:16.266330+0000 mgr.smithi006.odexej (mgr.14176) 422 : audit [DBG] from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:19.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:18 smithi006 bash[26022]: cluster 2023-11-17T05:43:17.815709+0000 mgr.smithi006.odexej (mgr.14176) 423 : cluster [DBG] pgmap v328: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:19.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:18 smithi156 bash[29655]: cluster 2023-11-17T05:43:17.815709+0000 mgr.smithi006.odexej (mgr.14176) 423 : cluster [DBG] pgmap v328: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:20.941 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:20.941 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:21.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:20 smithi006 bash[26022]: cluster 2023-11-17T05:43:19.817096+0000 mgr.smithi006.odexej (mgr.14176) 424 : cluster [DBG] pgmap v329: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:21.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:20 smithi156 bash[29655]: cluster 2023-11-17T05:43:19.817096+0000 mgr.smithi006.odexej (mgr.14176) 424 : cluster [DBG] pgmap v329: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:21.583 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:22.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:21 smithi006 bash[26022]: audit 2023-11-17T05:43:20.929369+0000 mgr.smithi006.odexej (mgr.14176) 425 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:22.414 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:21 smithi156 bash[29655]: audit 2023-11-17T05:43:20.929369+0000 mgr.smithi006.odexej (mgr.14176) 425 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:22.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:23.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:22 smithi006 bash[26022]: cluster 2023-11-17T05:43:21.818108+0000 mgr.smithi006.odexej (mgr.14176) 426 : cluster [DBG] pgmap v330: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:23.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:22 smithi156 bash[29655]: cluster 2023-11-17T05:43:21.818108+0000 mgr.smithi006.odexej (mgr.14176) 426 : cluster [DBG] pgmap v330: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:25.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:24 smithi006 bash[26022]: cluster 2023-11-17T05:43:23.819372+0000 mgr.smithi006.odexej (mgr.14176) 427 : cluster [DBG] pgmap v331: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:25.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:24 smithi156 bash[29655]: cluster 2023-11-17T05:43:23.819372+0000 mgr.smithi006.odexej (mgr.14176) 427 : cluster [DBG] pgmap v331: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:25.699 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:25.699 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:26.338 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:27.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:26 smithi156 bash[29655]: audit 2023-11-17T05:43:25.689196+0000 mgr.smithi006.odexej (mgr.14176) 428 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:27.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:26 smithi156 bash[29655]: cluster 2023-11-17T05:43:25.820521+0000 mgr.smithi006.odexej (mgr.14176) 429 : cluster [DBG] pgmap v332: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:27.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:27.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:26 smithi006 bash[26022]: audit 2023-11-17T05:43:25.689196+0000 mgr.smithi006.odexej (mgr.14176) 428 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:27.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:26 smithi006 bash[26022]: cluster 2023-11-17T05:43:25.820521+0000 mgr.smithi006.odexej (mgr.14176) 429 : cluster [DBG] pgmap v332: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:29.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:28 smithi156 bash[29655]: cluster 2023-11-17T05:43:27.821675+0000 mgr.smithi006.odexej (mgr.14176) 430 : cluster [DBG] pgmap v333: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:28 smithi006 bash[26022]: cluster 2023-11-17T05:43:27.821675+0000 mgr.smithi006.odexej (mgr.14176) 430 : cluster [DBG] pgmap v333: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:30.462 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:30.462 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:31.066 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:31.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:30 smithi156 bash[29655]: cluster 2023-11-17T05:43:29.822963+0000 mgr.smithi006.odexej (mgr.14176) 431 : cluster [DBG] pgmap v334: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:31.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:30 smithi006 bash[26022]: cluster 2023-11-17T05:43:29.822963+0000 mgr.smithi006.odexej (mgr.14176) 431 : cluster [DBG] pgmap v334: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:32.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:32.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:31 smithi006 bash[26022]: audit 2023-11-17T05:43:30.447420+0000 mgr.smithi006.odexej (mgr.14176) 432 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:32.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:31 smithi156 bash[29655]: audit 2023-11-17T05:43:30.447420+0000 mgr.smithi006.odexej (mgr.14176) 432 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:33.236 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:32 smithi006 bash[26022]: cluster 2023-11-17T05:43:31.823970+0000 mgr.smithi006.odexej (mgr.14176) 433 : cluster [DBG] pgmap v335: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:33.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:32 smithi156 bash[29655]: cluster 2023-11-17T05:43:31.823970+0000 mgr.smithi006.odexej (mgr.14176) 433 : cluster [DBG] pgmap v335: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:35.034 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:35.035 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:35.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:34 smithi006 bash[26022]: cluster 2023-11-17T05:43:33.825231+0000 mgr.smithi006.odexej (mgr.14176) 434 : cluster [DBG] pgmap v336: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:35.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:34 smithi156 bash[29655]: cluster 2023-11-17T05:43:33.825231+0000 mgr.smithi006.odexej (mgr.14176) 434 : cluster [DBG] pgmap v336: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:35.644 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:36.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:37.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:36 smithi006 bash[26022]: audit 2023-11-17T05:43:35.022429+0000 mgr.smithi006.odexej (mgr.14176) 435 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:37.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:36 smithi006 bash[26022]: cluster 2023-11-17T05:43:35.826626+0000 mgr.smithi006.odexej (mgr.14176) 436 : cluster [DBG] pgmap v337: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:37.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:36 smithi156 bash[29655]: audit 2023-11-17T05:43:35.022429+0000 mgr.smithi006.odexej (mgr.14176) 435 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:37.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:36 smithi156 bash[29655]: cluster 2023-11-17T05:43:35.826626+0000 mgr.smithi006.odexej (mgr.14176) 436 : cluster [DBG] pgmap v337: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:39.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:38 smithi006 bash[26022]: cluster 2023-11-17T05:43:37.827735+0000 mgr.smithi006.odexej (mgr.14176) 437 : cluster [DBG] pgmap v338: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:39.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:38 smithi156 bash[29655]: cluster 2023-11-17T05:43:37.827735+0000 mgr.smithi006.odexej (mgr.14176) 437 : cluster [DBG] pgmap v338: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:39.654 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:39.655 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:40.257 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:41.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:41.271 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:40 smithi006 bash[26022]: audit 2023-11-17T05:43:39.644320+0000 mgr.smithi006.odexej (mgr.14176) 438 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:41.271 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:40 smithi006 bash[26022]: cluster 2023-11-17T05:43:39.829186+0000 mgr.smithi006.odexej (mgr.14176) 439 : cluster [DBG] pgmap v339: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:41.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:40 smithi156 bash[29655]: audit 2023-11-17T05:43:39.644320+0000 mgr.smithi006.odexej (mgr.14176) 438 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:41.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:40 smithi156 bash[29655]: cluster 2023-11-17T05:43:39.829186+0000 mgr.smithi006.odexej (mgr.14176) 439 : cluster [DBG] pgmap v339: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:42.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:42 smithi006 bash[26022]: cluster 2023-11-17T05:43:41.830219+0000 mgr.smithi006.odexej (mgr.14176) 440 : cluster [DBG] pgmap v340: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:42.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:42 smithi156 bash[29655]: cluster 2023-11-17T05:43:41.830219+0000 mgr.smithi006.odexej (mgr.14176) 440 : cluster [DBG] pgmap v340: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:44.188 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:44.188 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:44.835 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:45.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:44 smithi006 bash[26022]: cluster 2023-11-17T05:43:43.831311+0000 mgr.smithi006.odexej (mgr.14176) 441 : cluster [DBG] pgmap v341: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:45.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:44 smithi156 bash[29655]: cluster 2023-11-17T05:43:43.831311+0000 mgr.smithi006.odexej (mgr.14176) 441 : cluster [DBG] pgmap v341: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:43:45.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:46.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:45 smithi006 bash[26022]: audit 2023-11-17T05:43:44.170554+0000 mgr.smithi006.odexej (mgr.14176) 442 : audit [DBG] from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:46.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:45 smithi156 bash[29655]: audit 2023-11-17T05:43:44.170554+0000 mgr.smithi006.odexej (mgr.14176) 442 : audit [DBG] from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:47.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:46 smithi006 bash[26022]: cluster 2023-11-17T05:43:45.832675+0000 mgr.smithi006.odexej (mgr.14176) 443 : cluster [DBG] pgmap v342: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:47.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:46 smithi156 bash[29655]: cluster 2023-11-17T05:43:45.832675+0000 mgr.smithi006.odexej (mgr.14176) 443 : cluster [DBG] pgmap v342: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:48.870 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:48.870 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:49.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:48 smithi006 bash[26022]: cluster 2023-11-17T05:43:47.833682+0000 mgr.smithi006.odexej (mgr.14176) 444 : cluster [DBG] pgmap v343: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:49.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:48 smithi156 bash[29655]: cluster 2023-11-17T05:43:47.833682+0000 mgr.smithi006.odexej (mgr.14176) 444 : cluster [DBG] pgmap v343: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:49.472 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:50.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:49 smithi006 bash[26022]: audit 2023-11-17T05:43:48.854537+0000 mgr.smithi006.odexej (mgr.14176) 445 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:50.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:49 smithi156 bash[29655]: audit 2023-11-17T05:43:48.854537+0000 mgr.smithi006.odexej (mgr.14176) 445 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:43:50.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:43:51.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:50 smithi156 bash[29655]: cluster 2023-11-17T05:43:49.835024+0000 mgr.smithi006.odexej (mgr.14176) 446 : cluster [DBG] pgmap v344: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:51.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:50 smithi006 bash[26022]: cluster 2023-11-17T05:43:49.835024+0000 mgr.smithi006.odexej (mgr.14176) 446 : cluster [DBG] pgmap v344: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:52.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:52 smithi006 bash[26022]: cluster 2023-11-17T05:43:51.836178+0000 mgr.smithi006.odexej (mgr.14176) 447 : cluster [DBG] pgmap v345: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:52 smithi156 bash[29655]: cluster 2023-11-17T05:43:51.836178+0000 mgr.smithi006.odexej (mgr.14176) 447 : cluster [DBG] pgmap v345: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:43:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:43:53.585 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:41:51.318880Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hhjvsr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:43:54.167 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:43:55.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:54 smithi006 bash[26022]: audit 2023-11-17T05:43:53.567458+0000 mgr.smithi006.odexej (mgr.14176) 448 : audit [DBG] from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:04.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:54 smithi006 bash[26022]: cluster 2023-11-17T05:43:53.837414+0000 mgr.smithi006.odexej (mgr.14176) 449 : cluster [DBG] pgmap v346: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:44:04.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:44:04.168 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:54 smithi156 bash[29655]: audit 2023-11-17T05:43:53.567458+0000 mgr.smithi006.odexej (mgr.14176) 448 : audit [DBG] from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:04.168 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:54 smithi156 bash[29655]: cluster 2023-11-17T05:43:53.837414+0000 mgr.smithi006.odexej (mgr.14176) 449 : cluster [DBG] pgmap v346: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:56 smithi156 bash[29655]: cluster 2023-11-17T05:43:55.838746+0000 mgr.smithi006.odexej (mgr.14176) 450 : cluster [DBG] pgmap v347: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:58 smithi156 bash[29655]: cluster 2023-11-17T05:43:57.839793+0000 mgr.smithi006.odexej (mgr.14176) 451 : cluster [DBG] pgmap v348: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:58 smithi156 bash[29655]: cluster 2023-11-17T05:43:57.923445+0000 mon.smithi006 (mon.0) 733 : cluster [DBG] mgrmap e21: smithi006.odexej(active, since 10m), standbys: smithi156.fphsnp 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:43:59 smithi156 bash[29655]: audit 2023-11-17T05:43:59.809052+0000 mon.smithi006 (mon.0) 734 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: cluster 2023-11-17T05:43:59.841089+0000 mgr.smithi006.odexej (mgr.14176) 452 : cluster [DBG] pgmap v349: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 170 B/s wr, 0 op/s 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: audit 2023-11-17T05:44:00.115147+0000 mon.smithi006 (mon.0) 735 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: audit 2023-11-17T05:44:00.116736+0000 mon.smithi006 (mon.0) 736 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: audit 2023-11-17T05:44:00.125579+0000 mon.smithi006 (mon.0) 737 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: audit 2023-11-17T05:44:00.135840+0000 mon.smithi006 (mon.0) 738 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:00 smithi156 bash[29655]: audit 2023-11-17T05:44:00.148381+0000 mon.smithi006 (mon.0) 739 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:01 smithi156 bash[29655]: cluster 2023-11-17T05:44:00.126608+0000 mgr.smithi006.odexej (mgr.14176) 453 : cluster [DBG] pgmap v350: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 198 B/s wr, 0 op/s 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:01 smithi156 bash[29655]: cephadm 2023-11-17T05:44:00.151484+0000 mgr.smithi006.odexej (mgr.14176) 454 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.nyrfwl on smithi156 2023-11-17T05:44:04.169 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:01 smithi156 bash[29655]: cluster 2023-11-17T05:44:01.121814+0000 mon.smithi006 (mon.0) 740 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:44:04.170 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:01 smithi156 bash[29655]: cluster 2023-11-17T05:44:01.121910+0000 mon.smithi006 (mon.0) 741 : cluster [INF] Cluster is now healthy 2023-11-17T05:44:04.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:56 smithi006 bash[26022]: cluster 2023-11-17T05:43:55.838746+0000 mgr.smithi006.odexej (mgr.14176) 450 : cluster [DBG] pgmap v347: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:04.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:58 smithi006 bash[26022]: cluster 2023-11-17T05:43:57.839793+0000 mgr.smithi006.odexej (mgr.14176) 451 : cluster [DBG] pgmap v348: 97 pgs: 97 active+clean; 3.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:04.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:58 smithi006 bash[26022]: cluster 2023-11-17T05:43:57.923445+0000 mon.smithi006 (mon.0) 733 : cluster [DBG] mgrmap e21: smithi006.odexej(active, since 10m), standbys: smithi156.fphsnp 2023-11-17T05:44:04.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:43:59 smithi006 bash[26022]: audit 2023-11-17T05:43:59.809052+0000 mon.smithi006 (mon.0) 734 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:44:04.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: cluster 2023-11-17T05:43:59.841089+0000 mgr.smithi006.odexej (mgr.14176) 452 : cluster [DBG] pgmap v349: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 170 B/s wr, 0 op/s 2023-11-17T05:44:04.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: audit 2023-11-17T05:44:00.115147+0000 mon.smithi006 (mon.0) 735 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:44:04.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: audit 2023-11-17T05:44:00.116736+0000 mon.smithi006 (mon.0) 736 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:44:04.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: audit 2023-11-17T05:44:00.125579+0000 mon.smithi006 (mon.0) 737 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:04.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: audit 2023-11-17T05:44:00.135840+0000 mon.smithi006 (mon.0) 738 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:00 smithi006 bash[26022]: audit 2023-11-17T05:44:00.148381+0000 mon.smithi006 (mon.0) 739 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:01 smithi006 bash[26022]: cluster 2023-11-17T05:44:00.126608+0000 mgr.smithi006.odexej (mgr.14176) 453 : cluster [DBG] pgmap v350: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 198 B/s wr, 0 op/s 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:01 smithi006 bash[26022]: cephadm 2023-11-17T05:44:00.151484+0000 mgr.smithi006.odexej (mgr.14176) 454 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.nyrfwl on smithi156 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:01 smithi006 bash[26022]: cluster 2023-11-17T05:44:01.121814+0000 mon.smithi006 (mon.0) 740 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:01 smithi006 bash[26022]: cluster 2023-11-17T05:44:01.121910+0000 mon.smithi006 (mon.0) 741 : cluster [INF] Cluster is now healthy 2023-11-17T05:44:04.523 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:03 smithi006 bash[26022]: cluster 2023-11-17T05:44:02.127378+0000 mgr.smithi006.odexej (mgr.14176) 455 : cluster [DBG] pgmap v351: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 198 B/s wr, 0 op/s 2023-11-17T05:44:04.524 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:03 smithi006 bash[26022]: audit 2023-11-17T05:44:03.877053+0000 mon.smithi006 (mon.0) 742 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.nyrfwl"}]: dispatch 2023-11-17T05:44:04.525 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:03 smithi156 bash[29655]: cluster 2023-11-17T05:44:02.127378+0000 mgr.smithi006.odexej (mgr.14176) 455 : cluster [DBG] pgmap v351: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 198 B/s wr, 0 op/s 2023-11-17T05:44:04.525 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:03 smithi156 bash[29655]: audit 2023-11-17T05:44:03.877053+0000 mon.smithi006 (mon.0) 742 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.nyrfwl"}]: dispatch 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: cephadm 2023-11-17T05:44:03.875924+0000 mgr.smithi006.odexej (mgr.14176) 456 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:05.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:06.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:06.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: cephadm 2023-11-17T05:44:03.876435+0000 mgr.smithi006.odexej (mgr.14176) 457 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: cephadm 2023-11-17T05:44:03.878868+0000 mgr.smithi006.odexej (mgr.14176) 458 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: cephadm 2023-11-17T05:44:03.883866+0000 mgr.smithi006.odexej (mgr.14176) 459 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.keksmh on smithi006 2023-11-17T05:44:06.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:04 smithi006 bash[26022]: audit 2023-11-17T05:44:04.706236+0000 mon.smithi006 (mon.0) 743 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:05 smithi006 bash[26022]: cluster 2023-11-17T05:44:04.128061+0000 mgr.smithi006.odexej (mgr.14176) 460 : cluster [DBG] pgmap v352: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 199 B/s wr, 0 op/s 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: cephadm 2023-11-17T05:44:03.875924+0000 mgr.smithi006.odexej (mgr.14176) 456 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:07.376 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.377 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: cephadm 2023-11-17T05:44:03.876435+0000 mgr.smithi006.odexej (mgr.14176) 457 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: cephadm 2023-11-17T05:44:03.878868+0000 mgr.smithi006.odexej (mgr.14176) 458 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.378 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:07.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl 2023-11-17T05:44:07.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.nyrfwl ... 2023-11-17T05:44:07.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:07.379 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:07.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:07.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:07.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: cephadm 2023-11-17T05:44:03.883866+0000 mgr.smithi006.odexej (mgr.14176) 459 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.keksmh on smithi006 2023-11-17T05:44:07.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:04 smithi156 bash[29655]: audit 2023-11-17T05:44:04.706236+0000 mon.smithi006 (mon.0) 743 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:07.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:05 smithi156 bash[29655]: cluster 2023-11-17T05:44:04.128061+0000 mgr.smithi006.odexej (mgr.14176) 460 : cluster [DBG] pgmap v352: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 199 B/s wr, 0 op/s 2023-11-17T05:44:07.638 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:44:07.638 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:55.688996Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qimxpq on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qimxpq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qimxpq\nDeploy daemon haproxy.nfs.foo.smithi156.qimxpq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.796256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.attjol on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-attjol\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.attjol\nDeploy daemon haproxy.nfs.foo.smithi006.attjol ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:42:59.800118Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.uvmatt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:44:08.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:07 smithi006 bash[26022]: cluster 2023-11-17T05:44:06.129198+0000 mgr.smithi006.odexej (mgr.14176) 461 : cluster [DBG] pgmap v353: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 199 B/s wr, 0 op/s 2023-11-17T05:44:08.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:07 smithi006 bash[26022]: audit 2023-11-17T05:44:07.636889+0000 mon.smithi006 (mon.0) 744 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.keksmh"}]: dispatch 2023-11-17T05:44:08.306 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:44:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:07 smithi156 bash[29655]: cluster 2023-11-17T05:44:06.129198+0000 mgr.smithi006.odexej (mgr.14176) 461 : cluster [DBG] pgmap v353: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 199 B/s wr, 0 op/s 2023-11-17T05:44:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:07 smithi156 bash[29655]: audit 2023-11-17T05:44:07.636889+0000 mon.smithi006 (mon.0) 744 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.keksmh"}]: dispatch 2023-11-17T05:44:09.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:44:09.319 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: audit 2023-11-17T05:44:07.628144+0000 mgr.smithi006.odexej (mgr.14176) 462 : audit [DBG] from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:09.319 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cephadm 2023-11-17T05:44:07.636077+0000 mgr.smithi006.odexej (mgr.14176) 463 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.319 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.319 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:44:09.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cephadm 2023-11-17T05:44:07.636353+0000 mgr.smithi006.odexej (mgr.14176) 464 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cephadm 2023-11-17T05:44:07.638130+0000 mgr.smithi006.odexej (mgr.14176) 465 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.322 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cephadm 2023-11-17T05:44:07.639963+0000 mgr.smithi006.odexej (mgr.14176) 466 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cephadm 2023-11-17T05:44:07.641488+0000 mgr.smithi006.odexej (mgr.14176) 467 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cluster 2023-11-17T05:44:07.642650+0000 mgr.smithi006.odexej (mgr.14176) 468 : cluster [DBG] pgmap v354: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 208 B/s wr, 0 op/s 2023-11-17T05:44:09.323 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:08 smithi006 bash[26022]: cluster 2023-11-17T05:44:07.951846+0000 mon.smithi006 (mon.0) 745 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:44:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: audit 2023-11-17T05:44:07.628144+0000 mgr.smithi006.odexej (mgr.14176) 462 : audit [DBG] from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cephadm 2023-11-17T05:44:07.636077+0000 mgr.smithi006.odexej (mgr.14176) 463 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cephadm 2023-11-17T05:44:07.636353+0000 mgr.smithi006.odexej (mgr.14176) 464 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cephadm 2023-11-17T05:44:07.638130+0000 mgr.smithi006.odexej (mgr.14176) 465 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.keksmh ... 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cephadm 2023-11-17T05:44:07.639963+0000 mgr.smithi006.odexej (mgr.14176) 466 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:44:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cephadm 2023-11-17T05:44:07.641488+0000 mgr.smithi006.odexej (mgr.14176) 467 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:44:09.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cluster 2023-11-17T05:44:07.642650+0000 mgr.smithi006.odexej (mgr.14176) 468 : cluster [DBG] pgmap v354: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 208 B/s wr, 0 op/s 2023-11-17T05:44:09.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:08 smithi156 bash[29655]: cluster 2023-11-17T05:44:07.951846+0000 mon.smithi006 (mon.0) 745 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:44:11.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:10 smithi006 bash[26022]: cluster 2023-11-17T05:44:09.644086+0000 mgr.smithi006.odexej (mgr.14176) 469 : cluster [DBG] pgmap v355: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:44:11.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:10 smithi006 bash[26022]: audit 2023-11-17T05:44:10.195658+0000 mon.smithi006 (mon.0) 746 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:11.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:10 smithi156 bash[29655]: cluster 2023-11-17T05:44:09.644086+0000 mgr.smithi006.odexej (mgr.14176) 469 : cluster [DBG] pgmap v355: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:44:11.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:10 smithi156 bash[29655]: audit 2023-11-17T05:44:10.195658+0000 mon.smithi006 (mon.0) 746 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:12.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:44:12.435 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:44:12.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:12 smithi006 bash[26022]: cluster 2023-11-17T05:44:11.645138+0000 mgr.smithi006.odexej (mgr.14176) 470 : cluster [DBG] pgmap v356: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:12.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:12 smithi156 bash[29655]: cluster 2023-11-17T05:44:11.645138+0000 mgr.smithi006.odexej (mgr.14176) 470 : cluster [DBG] pgmap v356: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:13.060 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:44:13.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:13 smithi156 bash[29655]: audit 2023-11-17T05:44:12.418139+0000 mgr.smithi006.odexej (mgr.14176) 471 : audit [DBG] from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:13.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:13 smithi006 bash[26022]: audit 2023-11-17T05:44:12.418139+0000 mgr.smithi006.odexej (mgr.14176) 471 : audit [DBG] from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:14.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:44:14.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:14 smithi006 bash[26022]: cluster 2023-11-17T05:44:13.646327+0000 mgr.smithi006.odexej (mgr.14176) 472 : cluster [DBG] pgmap v357: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:44:14.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:14 smithi156 bash[29655]: cluster 2023-11-17T05:44:13.646327+0000 mgr.smithi006.odexej (mgr.14176) 472 : cluster [DBG] pgmap v357: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:44:17.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:16 smithi006 bash[26022]: cluster 2023-11-17T05:44:15.647871+0000 mgr.smithi006.odexej (mgr.14176) 473 : cluster [DBG] pgmap v358: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:17.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:44:17.179 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:44:17.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:16 smithi156 bash[29655]: cluster 2023-11-17T05:44:15.647871+0000 mgr.smithi006.odexej (mgr.14176) 473 : cluster [DBG] pgmap v358: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:17.782 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:44:18.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:44:19.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:18 smithi006 bash[26022]: audit 2023-11-17T05:44:17.169218+0000 mgr.smithi006.odexej (mgr.14176) 474 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:19.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:18 smithi006 bash[26022]: cluster 2023-11-17T05:44:17.649018+0000 mgr.smithi006.odexej (mgr.14176) 475 : cluster [DBG] pgmap v359: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:19.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:18 smithi156 bash[29655]: audit 2023-11-17T05:44:17.169218+0000 mgr.smithi006.odexej (mgr.14176) 474 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:19.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:18 smithi156 bash[29655]: cluster 2023-11-17T05:44:17.649018+0000 mgr.smithi006.odexej (mgr.14176) 475 : cluster [DBG] pgmap v359: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:44:21.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:20 smithi006 bash[26022]: cluster 2023-11-17T05:44:19.650448+0000 mgr.smithi006.odexej (mgr.14176) 476 : cluster [DBG] pgmap v360: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:21.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:20 smithi006 bash[26022]: audit 2023-11-17T05:44:19.704968+0000 mon.smithi006 (mon.0) 747 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:21.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:20 smithi156 bash[29655]: cluster 2023-11-17T05:44:19.650448+0000 mgr.smithi006.odexej (mgr.14176) 476 : cluster [DBG] pgmap v360: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:21.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:20 smithi156 bash[29655]: audit 2023-11-17T05:44:19.704968+0000 mon.smithi006 (mon.0) 747 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:44:21.938 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:44:27.521 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:44:27.608 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:22 smithi156 bash[29655]: cluster 2023-11-17T05:44:21.651448+0000 mgr.smithi006.odexej (mgr.14176) 477 : cluster [DBG] pgmap v361: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:27.608 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:22 smithi156 bash[29655]: audit 2023-11-17T05:44:21.921037+0000 mgr.smithi006.odexej (mgr.14176) 478 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:27.609 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:24 smithi156 bash[29655]: cluster 2023-11-17T05:44:23.652726+0000 mgr.smithi006.odexej (mgr.14176) 479 : cluster [DBG] pgmap v362: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:44:27.609 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:26 smithi156 bash[29655]: cluster 2023-11-17T05:44:25.654135+0000 mgr.smithi006.odexej (mgr.14176) 480 : cluster [DBG] pgmap v363: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:27.610 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:44:27.611 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:22 smithi006 bash[26022]: cluster 2023-11-17T05:44:21.651448+0000 mgr.smithi006.odexej (mgr.14176) 477 : cluster [DBG] pgmap v361: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:27.611 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:22 smithi006 bash[26022]: audit 2023-11-17T05:44:21.921037+0000 mgr.smithi006.odexej (mgr.14176) 478 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:44:27.611 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:24 smithi006 bash[26022]: cluster 2023-11-17T05:44:23.652726+0000 mgr.smithi006.odexej (mgr.14176) 479 : cluster [DBG] pgmap v362: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:44:27.611 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:26 smithi006 bash[26022]: cluster 2023-11-17T05:44:25.654135+0000 mgr.smithi006.odexej (mgr.14176) 480 : cluster [DBG] pgmap v363: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:28.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:44:28.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:28 smithi006 bash[26022]: cluster 2023-11-17T05:44:27.655151+0000 mgr.smithi006.odexej (mgr.14176) 481 : cluster [DBG] pgmap v364: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:29.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:28 smithi156 bash[29655]: cluster 2023-11-17T05:44:27.655151+0000 mgr.smithi006.odexej (mgr.14176) 481 : cluster [DBG] pgmap v364: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:44:31.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:30 smithi006 bash[26022]: cluster 2023-11-17T05:44:29.656644+0000 mgr.smithi006.odexej (mgr.14176) 482 : cluster [DBG] pgmap v365: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.612 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:01.612 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:01.613 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:01.665 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:30 smithi156 bash[29655]: cluster 2023-11-17T05:44:29.656644+0000 mgr.smithi006.odexej (mgr.14176) 482 : cluster [DBG] pgmap v365: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.665 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:32 smithi156 bash[29655]: cluster 2023-11-17T05:44:31.657715+0000 mgr.smithi006.odexej (mgr.14176) 483 : cluster [DBG] pgmap v366: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:32 smithi156 bash[29655]: audit 2023-11-17T05:44:31.720829+0000 mgr.smithi006.odexej (mgr.14176) 484 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:01.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:34 smithi156 bash[29655]: cluster 2023-11-17T05:44:33.658942+0000 mgr.smithi006.odexej (mgr.14176) 485 : cluster [DBG] pgmap v367: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:36 smithi156 bash[29655]: cluster 2023-11-17T05:44:35.660287+0000 mgr.smithi006.odexej (mgr.14176) 486 : cluster [DBG] pgmap v368: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:38 smithi156 bash[29655]: cluster 2023-11-17T05:44:37.661346+0000 mgr.smithi006.odexej (mgr.14176) 487 : cluster [DBG] pgmap v369: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:40 smithi156 bash[29655]: cluster 2023-11-17T05:44:39.662779+0000 mgr.smithi006.odexej (mgr.14176) 488 : cluster [DBG] pgmap v370: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:42 smithi156 bash[29655]: cluster 2023-11-17T05:44:41.663881+0000 mgr.smithi006.odexej (mgr.14176) 489 : cluster [DBG] pgmap v371: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:44 smithi156 bash[29655]: cluster 2023-11-17T05:44:43.665029+0000 mgr.smithi006.odexej (mgr.14176) 490 : cluster [DBG] pgmap v372: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:46 smithi156 bash[29655]: cluster 2023-11-17T05:44:45.666522+0000 mgr.smithi006.odexej (mgr.14176) 491 : cluster [DBG] pgmap v373: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:48 smithi156 bash[29655]: cluster 2023-11-17T05:44:47.667643+0000 mgr.smithi006.odexej (mgr.14176) 492 : cluster [DBG] pgmap v374: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:50 smithi156 bash[29655]: cluster 2023-11-17T05:44:49.669026+0000 mgr.smithi006.odexej (mgr.14176) 493 : cluster [DBG] pgmap v375: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:52 smithi156 bash[29655]: cluster 2023-11-17T05:44:51.670015+0000 mgr.smithi006.odexej (mgr.14176) 494 : cluster [DBG] pgmap v376: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:54 smithi156 bash[29655]: cluster 2023-11-17T05:44:53.671176+0000 mgr.smithi006.odexej (mgr.14176) 495 : cluster [DBG] pgmap v377: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:56 smithi156 bash[29655]: cluster 2023-11-17T05:44:55.672524+0000 mgr.smithi006.odexej (mgr.14176) 496 : cluster [DBG] pgmap v378: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.699 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:44:58 smithi156 bash[29655]: cluster 2023-11-17T05:44:57.673507+0000 mgr.smithi006.odexej (mgr.14176) 497 : cluster [DBG] pgmap v379: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.699 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:00 smithi156 bash[29655]: cluster 2023-11-17T05:44:59.674912+0000 mgr.smithi006.odexej (mgr.14176) 498 : cluster [DBG] pgmap v380: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.699 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:32 smithi006 bash[26022]: cluster 2023-11-17T05:44:31.657715+0000 mgr.smithi006.odexej (mgr.14176) 483 : cluster [DBG] pgmap v366: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.699 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:32 smithi006 bash[26022]: audit 2023-11-17T05:44:31.720829+0000 mgr.smithi006.odexej (mgr.14176) 484 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:01.703 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:34 smithi006 bash[26022]: cluster 2023-11-17T05:44:33.658942+0000 mgr.smithi006.odexej (mgr.14176) 485 : cluster [DBG] pgmap v367: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:36 smithi006 bash[26022]: cluster 2023-11-17T05:44:35.660287+0000 mgr.smithi006.odexej (mgr.14176) 486 : cluster [DBG] pgmap v368: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.924 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:38 smithi006 bash[26022]: cluster 2023-11-17T05:44:37.661346+0000 mgr.smithi006.odexej (mgr.14176) 487 : cluster [DBG] pgmap v369: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:40 smithi006 bash[26022]: cluster 2023-11-17T05:44:39.662779+0000 mgr.smithi006.odexej (mgr.14176) 488 : cluster [DBG] pgmap v370: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:42 smithi006 bash[26022]: cluster 2023-11-17T05:44:41.663881+0000 mgr.smithi006.odexej (mgr.14176) 489 : cluster [DBG] pgmap v371: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:44 smithi006 bash[26022]: cluster 2023-11-17T05:44:43.665029+0000 mgr.smithi006.odexej (mgr.14176) 490 : cluster [DBG] pgmap v372: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:46 smithi006 bash[26022]: cluster 2023-11-17T05:44:45.666522+0000 mgr.smithi006.odexej (mgr.14176) 491 : cluster [DBG] pgmap v373: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:48 smithi006 bash[26022]: cluster 2023-11-17T05:44:47.667643+0000 mgr.smithi006.odexej (mgr.14176) 492 : cluster [DBG] pgmap v374: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:50 smithi006 bash[26022]: cluster 2023-11-17T05:44:49.669026+0000 mgr.smithi006.odexej (mgr.14176) 493 : cluster [DBG] pgmap v375: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:52 smithi006 bash[26022]: cluster 2023-11-17T05:44:51.670015+0000 mgr.smithi006.odexej (mgr.14176) 494 : cluster [DBG] pgmap v376: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:54 smithi006 bash[26022]: cluster 2023-11-17T05:44:53.671176+0000 mgr.smithi006.odexej (mgr.14176) 495 : cluster [DBG] pgmap v377: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:01.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:56 smithi006 bash[26022]: cluster 2023-11-17T05:44:55.672524+0000 mgr.smithi006.odexej (mgr.14176) 496 : cluster [DBG] pgmap v378: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.962 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:44:58 smithi006 bash[26022]: cluster 2023-11-17T05:44:57.673507+0000 mgr.smithi006.odexej (mgr.14176) 497 : cluster [DBG] pgmap v379: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:01.962 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:00 smithi006 bash[26022]: cluster 2023-11-17T05:44:59.674912+0000 mgr.smithi006.odexej (mgr.14176) 498 : cluster [DBG] pgmap v380: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:02.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:02.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:02 smithi006 bash[26022]: cluster 2023-11-17T05:45:01.675976+0000 mgr.smithi006.odexej (mgr.14176) 499 : cluster [DBG] pgmap v381: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:02.698 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:02 smithi156 bash[29655]: cluster 2023-11-17T05:45:01.675976+0000 mgr.smithi006.odexej (mgr.14176) 499 : cluster [DBG] pgmap v381: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:05.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:04 smithi006 bash[26022]: cluster 2023-11-17T05:45:03.677141+0000 mgr.smithi006.odexej (mgr.14176) 500 : cluster [DBG] pgmap v382: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:05.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:04 smithi156 bash[29655]: cluster 2023-11-17T05:45:03.677141+0000 mgr.smithi006.odexej (mgr.14176) 500 : cluster [DBG] pgmap v382: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:05.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:05.806 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:06.410 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:07.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:06 smithi006 bash[26022]: cluster 2023-11-17T05:45:05.678302+0000 mgr.smithi006.odexej (mgr.14176) 501 : cluster [DBG] pgmap v383: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:07.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:06 smithi006 bash[26022]: audit 2023-11-17T05:45:05.790665+0000 mgr.smithi006.odexej (mgr.14176) 502 : audit [DBG] from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:07.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:06 smithi156 bash[29655]: cluster 2023-11-17T05:45:05.678302+0000 mgr.smithi006.odexej (mgr.14176) 501 : cluster [DBG] pgmap v383: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:07.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:06 smithi156 bash[29655]: audit 2023-11-17T05:45:05.790665+0000 mgr.smithi006.odexej (mgr.14176) 502 : audit [DBG] from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:07.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:08.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:07 smithi006 bash[26022]: audit 2023-11-17T05:45:07.645392+0000 mon.smithi006 (mon.0) 748 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:45:08.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:07 smithi156 bash[29655]: audit 2023-11-17T05:45:07.645392+0000 mon.smithi006 (mon.0) 748 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:45:09.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: cluster 2023-11-17T05:45:07.679071+0000 mgr.smithi006.odexej (mgr.14176) 503 : cluster [DBG] pgmap v384: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:07.947682+0000 mon.smithi006 (mon.0) 749 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:08.232132+0000 mon.smithi006 (mon.0) 750 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:08.234155+0000 mon.smithi006 (mon.0) 751 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:08.245052+0000 mon.smithi006 (mon.0) 752 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:08.253910+0000 mon.smithi006 (mon.0) 753 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:45:09.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:08 smithi006 bash[26022]: audit 2023-11-17T05:45:08.272770+0000 mon.smithi006 (mon.0) 754 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: cluster 2023-11-17T05:45:07.679071+0000 mgr.smithi006.odexej (mgr.14176) 503 : cluster [DBG] pgmap v384: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:07.947682+0000 mon.smithi006 (mon.0) 749 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:08.232132+0000 mon.smithi006 (mon.0) 750 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:45:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:08.234155+0000 mon.smithi006 (mon.0) 751 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:45:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:08.245052+0000 mon.smithi006 (mon.0) 752 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:08.253910+0000 mon.smithi006 (mon.0) 753 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:45:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:08 smithi156 bash[29655]: audit 2023-11-17T05:45:08.272770+0000 mon.smithi006 (mon.0) 754 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:10.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:09 smithi006 bash[26022]: cluster 2023-11-17T05:45:08.246344+0000 mgr.smithi006.odexej (mgr.14176) 504 : cluster [DBG] pgmap v385: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:10.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:09 smithi006 bash[26022]: cephadm 2023-11-17T05:45:08.277279+0000 mgr.smithi006.odexej (mgr.14176) 505 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dmllff on smithi156 2023-11-17T05:45:10.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:09 smithi006 bash[26022]: cluster 2023-11-17T05:45:09.240387+0000 mon.smithi006 (mon.0) 755 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:45:10.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:09 smithi006 bash[26022]: cluster 2023-11-17T05:45:09.240456+0000 mon.smithi006 (mon.0) 756 : cluster [INF] Cluster is now healthy 2023-11-17T05:45:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:09 smithi156 bash[29655]: cluster 2023-11-17T05:45:08.246344+0000 mgr.smithi006.odexej (mgr.14176) 504 : cluster [DBG] pgmap v385: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:09 smithi156 bash[29655]: cephadm 2023-11-17T05:45:08.277279+0000 mgr.smithi006.odexej (mgr.14176) 505 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dmllff on smithi156 2023-11-17T05:45:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:09 smithi156 bash[29655]: cluster 2023-11-17T05:45:09.240387+0000 mon.smithi006 (mon.0) 755 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:45:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:09 smithi156 bash[29655]: cluster 2023-11-17T05:45:09.240456+0000 mon.smithi006 (mon.0) 756 : cluster [INF] Cluster is now healthy 2023-11-17T05:45:10.680 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:10.680 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:42:59.803574Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ejlrko on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:11.277 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:12.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:12.288 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:11 smithi006 bash[26022]: cluster 2023-11-17T05:45:10.247501+0000 mgr.smithi006.odexej (mgr.14176) 506 : cluster [DBG] pgmap v386: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:12.288 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:11 smithi006 bash[26022]: audit 2023-11-17T05:45:10.661924+0000 mgr.smithi006.odexej (mgr.14176) 507 : audit [DBG] from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:12.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:11 smithi156 bash[29655]: cluster 2023-11-17T05:45:10.247501+0000 mgr.smithi006.odexej (mgr.14176) 506 : cluster [DBG] pgmap v386: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:12.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:11 smithi156 bash[29655]: audit 2023-11-17T05:45:10.661924+0000 mgr.smithi006.odexej (mgr.14176) 507 : audit [DBG] from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:13.271 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:12 smithi006 bash[26022]: audit 2023-11-17T05:45:12.016639+0000 mon.smithi006 (mon.0) 757 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dmllff"}]: dispatch 2023-11-17T05:45:13.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:12 smithi156 bash[29655]: audit 2023-11-17T05:45:12.016639+0000 mon.smithi006 (mon.0) 757 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dmllff"}]: dispatch 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: cephadm 2023-11-17T05:45:12.015206+0000 mgr.smithi006.odexej (mgr.14176) 508 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: cephadm 2023-11-17T05:45:12.015813+0000 mgr.smithi006.odexej (mgr.14176) 509 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: cephadm 2023-11-17T05:45:12.018502+0000 mgr.smithi006.odexej (mgr.14176) 510 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: cephadm 2023-11-17T05:45:12.023800+0000 mgr.smithi006.odexej (mgr.14176) 511 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.wwldxr on smithi006 2023-11-17T05:45:14.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:13 smithi006 bash[26022]: cluster 2023-11-17T05:45:12.248141+0000 mgr.smithi006.odexej (mgr.14176) 512 : cluster [DBG] pgmap v387: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: cephadm 2023-11-17T05:45:12.015206+0000 mgr.smithi006.odexej (mgr.14176) 508 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: cephadm 2023-11-17T05:45:12.015813+0000 mgr.smithi006.odexej (mgr.14176) 509 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: cephadm 2023-11-17T05:45:12.018502+0000 mgr.smithi006.odexej (mgr.14176) 510 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dmllff ... 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: cephadm 2023-11-17T05:45:12.023800+0000 mgr.smithi006.odexej (mgr.14176) 511 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.wwldxr on smithi006 2023-11-17T05:45:14.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:13 smithi156 bash[29655]: cluster 2023-11-17T05:45:12.248141+0000 mgr.smithi006.odexej (mgr.14176) 512 : cluster [DBG] pgmap v387: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:15.287 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:15.288 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:03.878539Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.nyrfwl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-nyrfwl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.nyrfwl\nDeploy daemon haproxy.nfs.foo.smithi156.nyrfwl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.638006Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.keksmh on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-keksmh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.keksmh\nDeploy daemon haproxy.nfs.foo.smithi006.keksmh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:44:07.639861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fxmpnl on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:15.952 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:16.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:15 smithi006 bash[26022]: cluster 2023-11-17T05:45:14.248990+0000 mgr.smithi006.odexej (mgr.14176) 513 : cluster [DBG] pgmap v388: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:16.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:15 smithi006 bash[26022]: audit 2023-11-17T05:45:15.909730+0000 mon.smithi006 (mon.0) 758 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.wwldxr"}]: dispatch 2023-11-17T05:45:16.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:15 smithi156 bash[29655]: cluster 2023-11-17T05:45:14.248990+0000 mgr.smithi006.odexej (mgr.14176) 513 : cluster [DBG] pgmap v388: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:45:16.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:15 smithi156 bash[29655]: audit 2023-11-17T05:45:15.909730+0000 mon.smithi006 (mon.0) 758 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.wwldxr"}]: dispatch 2023-11-17T05:45:16.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:17.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: audit 2023-11-17T05:45:15.275982+0000 mgr.smithi006.odexej (mgr.14176) 514 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:17.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cephadm 2023-11-17T05:45:15.908896+0000 mgr.smithi006.odexej (mgr.14176) 515 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:45:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cephadm 2023-11-17T05:45:15.909191+0000 mgr.smithi006.odexej (mgr.14176) 516 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cephadm 2023-11-17T05:45:15.910828+0000 mgr.smithi006.odexej (mgr.14176) 517 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cephadm 2023-11-17T05:45:15.912985+0000 mgr.smithi006.odexej (mgr.14176) 518 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:45:17.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cephadm 2023-11-17T05:45:15.914838+0000 mgr.smithi006.odexej (mgr.14176) 519 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:45:17.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cluster 2023-11-17T05:45:15.915961+0000 mgr.smithi006.odexej (mgr.14176) 520 : cluster [DBG] pgmap v389: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T05:45:17.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:16 smithi006 bash[26022]: cluster 2023-11-17T05:45:15.982964+0000 mon.smithi006 (mon.0) 759 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:45:17.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: audit 2023-11-17T05:45:15.275982+0000 mgr.smithi006.odexej (mgr.14176) 514 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cephadm 2023-11-17T05:45:15.908896+0000 mgr.smithi006.odexej (mgr.14176) 515 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cephadm 2023-11-17T05:45:15.909191+0000 mgr.smithi006.odexej (mgr.14176) 516 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cephadm 2023-11-17T05:45:15.910828+0000 mgr.smithi006.odexej (mgr.14176) 517 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.wwldxr ... 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cephadm 2023-11-17T05:45:15.912985+0000 mgr.smithi006.odexej (mgr.14176) 518 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cephadm 2023-11-17T05:45:15.914838+0000 mgr.smithi006.odexej (mgr.14176) 519 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cluster 2023-11-17T05:45:15.915961+0000 mgr.smithi006.odexej (mgr.14176) 520 : cluster [DBG] pgmap v389: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T05:45:17.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:16 smithi156 bash[29655]: cluster 2023-11-17T05:45:15.982964+0000 mon.smithi006 (mon.0) 759 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:45:19.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:18 smithi006 bash[26022]: cluster 2023-11-17T05:45:17.916944+0000 mgr.smithi006.odexej (mgr.14176) 521 : cluster [DBG] pgmap v390: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T05:45:19.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:18 smithi156 bash[29655]: cluster 2023-11-17T05:45:17.916944+0000 mgr.smithi006.odexej (mgr.14176) 521 : cluster [DBG] pgmap v390: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T05:45:20.059 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:20.060 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:20.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:20 smithi006 bash[26022]: cluster 2023-11-17T05:45:19.918313+0000 mgr.smithi006.odexej (mgr.14176) 522 : cluster [DBG] pgmap v391: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:20.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:20 smithi156 bash[29655]: cluster 2023-11-17T05:45:19.918313+0000 mgr.smithi006.odexej (mgr.14176) 522 : cluster [DBG] pgmap v391: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:20.673 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:21.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:21 smithi006 bash[26022]: audit 2023-11-17T05:45:20.039172+0000 mgr.smithi006.odexej (mgr.14176) 523 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:21.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:21 smithi006 bash[26022]: audit 2023-11-17T05:45:20.248456+0000 mon.smithi006 (mon.0) 760 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:21.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:21.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:21 smithi156 bash[29655]: audit 2023-11-17T05:45:20.039172+0000 mgr.smithi006.odexej (mgr.14176) 523 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:21.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:21 smithi156 bash[29655]: audit 2023-11-17T05:45:20.248456+0000 mon.smithi006 (mon.0) 760 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:45:22.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:22 smithi006 bash[26022]: cluster 2023-11-17T05:45:21.919422+0000 mgr.smithi006.odexej (mgr.14176) 524 : cluster [DBG] pgmap v392: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:22.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:22 smithi156 bash[29655]: cluster 2023-11-17T05:45:21.919422+0000 mgr.smithi006.odexej (mgr.14176) 524 : cluster [DBG] pgmap v392: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:24.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:24.663 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:25.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:24 smithi006 bash[26022]: cluster 2023-11-17T05:45:23.920652+0000 mgr.smithi006.odexej (mgr.14176) 525 : cluster [DBG] pgmap v393: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T05:45:25.251 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:24 smithi156 bash[29655]: cluster 2023-11-17T05:45:23.920652+0000 mgr.smithi006.odexej (mgr.14176) 525 : cluster [DBG] pgmap v393: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T05:45:26.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:26.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:25 smithi006 bash[26022]: audit 2023-11-17T05:45:24.648028+0000 mgr.smithi006.odexej (mgr.14176) 526 : audit [DBG] from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:26.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:25 smithi156 bash[29655]: audit 2023-11-17T05:45:24.648028+0000 mgr.smithi006.odexej (mgr.14176) 526 : audit [DBG] from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:27.261 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:26 smithi006 bash[26022]: cluster 2023-11-17T05:45:25.922017+0000 mgr.smithi006.odexej (mgr.14176) 527 : cluster [DBG] pgmap v394: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:27.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:26 smithi156 bash[29655]: cluster 2023-11-17T05:45:25.922017+0000 mgr.smithi006.odexej (mgr.14176) 527 : cluster [DBG] pgmap v394: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:45:29.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:29 smithi006 bash[26022]: cluster 2023-11-17T05:45:27.923038+0000 mgr.smithi006.odexej (mgr.14176) 528 : cluster [DBG] pgmap v395: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:29.218 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:29.218 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:29.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:29 smithi156 bash[29655]: cluster 2023-11-17T05:45:27.923038+0000 mgr.smithi006.odexej (mgr.14176) 528 : cluster [DBG] pgmap v395: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:29.818 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:30.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:30 smithi006 bash[26022]: audit 2023-11-17T05:45:29.200494+0000 mgr.smithi006.odexej (mgr.14176) 529 : audit [DBG] from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:30.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:30 smithi006 bash[26022]: cluster 2023-11-17T05:45:29.924479+0000 mgr.smithi006.odexej (mgr.14176) 530 : cluster [DBG] pgmap v396: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:30 smithi156 bash[29655]: audit 2023-11-17T05:45:29.200494+0000 mgr.smithi006.odexej (mgr.14176) 529 : audit [DBG] from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:30 smithi156 bash[29655]: cluster 2023-11-17T05:45:29.924479+0000 mgr.smithi006.odexej (mgr.14176) 530 : cluster [DBG] pgmap v396: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:30.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:32.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:32 smithi006 bash[26022]: cluster 2023-11-17T05:45:31.925673+0000 mgr.smithi006.odexej (mgr.14176) 531 : cluster [DBG] pgmap v397: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:32.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:32 smithi156 bash[29655]: cluster 2023-11-17T05:45:31.925673+0000 mgr.smithi006.odexej (mgr.14176) 531 : cluster [DBG] pgmap v397: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:34.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:34.006 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:34.593 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:35.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:34 smithi006 bash[26022]: cluster 2023-11-17T05:45:33.926860+0000 mgr.smithi006.odexej (mgr.14176) 532 : cluster [DBG] pgmap v398: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:35.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:34 smithi006 bash[26022]: audit 2023-11-17T05:45:33.991288+0000 mgr.smithi006.odexej (mgr.14176) 533 : audit [DBG] from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:35.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:34 smithi156 bash[29655]: cluster 2023-11-17T05:45:33.926860+0000 mgr.smithi006.odexej (mgr.14176) 532 : cluster [DBG] pgmap v398: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:35.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:34 smithi156 bash[29655]: audit 2023-11-17T05:45:33.991288+0000 mgr.smithi006.odexej (mgr.14176) 533 : audit [DBG] from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:35.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:37.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:36 smithi006 bash[26022]: cluster 2023-11-17T05:45:35.928275+0000 mgr.smithi006.odexej (mgr.14176) 534 : cluster [DBG] pgmap v399: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:37.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:36 smithi156 bash[29655]: cluster 2023-11-17T05:45:35.928275+0000 mgr.smithi006.odexej (mgr.14176) 534 : cluster [DBG] pgmap v399: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:38.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:38.687 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:39.274 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:38 smithi006 bash[26022]: cluster 2023-11-17T05:45:37.929358+0000 mgr.smithi006.odexej (mgr.14176) 535 : cluster [DBG] pgmap v400: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:39.275 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:39.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:38 smithi156 bash[29655]: cluster 2023-11-17T05:45:37.929358+0000 mgr.smithi006.odexej (mgr.14176) 535 : cluster [DBG] pgmap v400: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:40.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:40.289 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:40 smithi006 bash[26022]: audit 2023-11-17T05:45:38.668678+0000 mgr.smithi006.odexej (mgr.14176) 536 : audit [DBG] from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:40.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:40 smithi156 bash[29655]: audit 2023-11-17T05:45:38.668678+0000 mgr.smithi006.odexej (mgr.14176) 536 : audit [DBG] from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:41.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:41 smithi006 bash[26022]: cluster 2023-11-17T05:45:39.930744+0000 mgr.smithi006.odexej (mgr.14176) 537 : cluster [DBG] pgmap v401: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:41.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:41 smithi156 bash[29655]: cluster 2023-11-17T05:45:39.930744+0000 mgr.smithi006.odexej (mgr.14176) 537 : cluster [DBG] pgmap v401: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:42.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:42 smithi006 bash[26022]: cluster 2023-11-17T05:45:41.931825+0000 mgr.smithi006.odexej (mgr.14176) 538 : cluster [DBG] pgmap v402: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:42.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:42 smithi156 bash[29655]: cluster 2023-11-17T05:45:41.931825+0000 mgr.smithi006.odexej (mgr.14176) 538 : cluster [DBG] pgmap v402: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:43.395 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:43.395 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:43.959 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:44.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:45.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:44 smithi006 bash[26022]: audit 2023-11-17T05:45:43.376710+0000 mgr.smithi006.odexej (mgr.14176) 539 : audit [DBG] from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:45.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:44 smithi006 bash[26022]: cluster 2023-11-17T05:45:43.933084+0000 mgr.smithi006.odexej (mgr.14176) 540 : cluster [DBG] pgmap v403: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:45.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:44 smithi156 bash[29655]: audit 2023-11-17T05:45:43.376710+0000 mgr.smithi006.odexej (mgr.14176) 539 : audit [DBG] from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:45.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:44 smithi156 bash[29655]: cluster 2023-11-17T05:45:43.933084+0000 mgr.smithi006.odexej (mgr.14176) 540 : cluster [DBG] pgmap v403: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:47.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:46 smithi006 bash[26022]: cluster 2023-11-17T05:45:45.934515+0000 mgr.smithi006.odexej (mgr.14176) 541 : cluster [DBG] pgmap v404: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:47.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:46 smithi156 bash[29655]: cluster 2023-11-17T05:45:45.934515+0000 mgr.smithi006.odexej (mgr.14176) 541 : cluster [DBG] pgmap v404: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:48.025 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:48.025 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:48.643 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:49.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:49 smithi006 bash[26022]: cluster 2023-11-17T05:45:47.935706+0000 mgr.smithi006.odexej (mgr.14176) 542 : cluster [DBG] pgmap v405: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:49.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:49 smithi156 bash[29655]: cluster 2023-11-17T05:45:47.935706+0000 mgr.smithi006.odexej (mgr.14176) 542 : cluster [DBG] pgmap v405: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:49.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:50.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:50 smithi006 bash[26022]: audit 2023-11-17T05:45:48.006117+0000 mgr.smithi006.odexej (mgr.14176) 543 : audit [DBG] from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:50.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:50 smithi006 bash[26022]: cluster 2023-11-17T05:45:49.937137+0000 mgr.smithi006.odexej (mgr.14176) 544 : cluster [DBG] pgmap v406: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:50.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:50 smithi156 bash[29655]: audit 2023-11-17T05:45:48.006117+0000 mgr.smithi006.odexej (mgr.14176) 543 : audit [DBG] from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:50.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:50 smithi156 bash[29655]: cluster 2023-11-17T05:45:49.937137+0000 mgr.smithi006.odexej (mgr.14176) 544 : cluster [DBG] pgmap v406: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:52.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:52 smithi006 bash[26022]: cluster 2023-11-17T05:45:51.938155+0000 mgr.smithi006.odexej (mgr.14176) 545 : cluster [DBG] pgmap v407: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:52.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:52 smithi156 bash[29655]: cluster 2023-11-17T05:45:51.938155+0000 mgr.smithi006.odexej (mgr.14176) 545 : cluster [DBG] pgmap v407: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:52.887 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:52.887 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:53.541 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:53.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:53 smithi156 bash[29655]: audit 2023-11-17T05:45:52.870835+0000 mgr.smithi006.odexej (mgr.14176) 546 : audit [DBG] from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:53.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:53 smithi006 bash[26022]: audit 2023-11-17T05:45:52.870835+0000 mgr.smithi006.odexej (mgr.14176) 546 : audit [DBG] from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:54.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:54.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:54 smithi006 bash[26022]: cluster 2023-11-17T05:45:53.939339+0000 mgr.smithi006.odexej (mgr.14176) 547 : cluster [DBG] pgmap v408: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:54.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:54 smithi156 bash[29655]: cluster 2023-11-17T05:45:53.939339+0000 mgr.smithi006.odexej (mgr.14176) 547 : cluster [DBG] pgmap v408: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:45:57.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:56 smithi006 bash[26022]: cluster 2023-11-17T05:45:55.940710+0000 mgr.smithi006.odexej (mgr.14176) 548 : cluster [DBG] pgmap v409: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:57.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:57 smithi156 bash[29655]: cluster 2023-11-17T05:45:55.940710+0000 mgr.smithi006.odexej (mgr.14176) 548 : cluster [DBG] pgmap v409: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:57.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:45:57.435 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:45:58.054 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:45:59.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:45:59.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:59 smithi006 bash[26022]: audit 2023-11-17T05:45:57.418518+0000 mgr.smithi006.odexej (mgr.14176) 549 : audit [DBG] from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:59.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:45:59 smithi006 bash[26022]: cluster 2023-11-17T05:45:57.941773+0000 mgr.smithi006.odexej (mgr.14176) 550 : cluster [DBG] pgmap v410: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:45:59.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:59 smithi156 bash[29655]: audit 2023-11-17T05:45:57.418518+0000 mgr.smithi006.odexej (mgr.14176) 549 : audit [DBG] from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:45:59.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:45:59 smithi156 bash[29655]: cluster 2023-11-17T05:45:57.941773+0000 mgr.smithi006.odexej (mgr.14176) 550 : cluster [DBG] pgmap v410: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:00.338 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:00 smithi006 bash[26022]: cluster 2023-11-17T05:45:59.943205+0000 mgr.smithi006.odexej (mgr.14176) 551 : cluster [DBG] pgmap v411: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:00 smithi156 bash[29655]: cluster 2023-11-17T05:45:59.943205+0000 mgr.smithi006.odexej (mgr.14176) 551 : cluster [DBG] pgmap v411: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:02.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:02.067 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:02.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:02 smithi006 bash[26022]: cluster 2023-11-17T05:46:01.943916+0000 mgr.smithi006.odexej (mgr.14176) 552 : cluster [DBG] pgmap v412: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:02.681 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:02.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:02 smithi156 bash[29655]: cluster 2023-11-17T05:46:01.943916+0000 mgr.smithi006.odexej (mgr.14176) 552 : cluster [DBG] pgmap v412: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:03.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:03 smithi156 bash[29655]: audit 2023-11-17T05:46:02.047760+0000 mgr.smithi006.odexej (mgr.14176) 553 : audit [DBG] from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:03.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:03 smithi006 bash[26022]: audit 2023-11-17T05:46:02.047760+0000 mgr.smithi006.odexej (mgr.14176) 553 : audit [DBG] from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:03.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:04.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:04 smithi006 bash[26022]: cluster 2023-11-17T05:46:03.945087+0000 mgr.smithi006.odexej (mgr.14176) 554 : cluster [DBG] pgmap v413: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:04.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:04 smithi156 bash[29655]: cluster 2023-11-17T05:46:03.945087+0000 mgr.smithi006.odexej (mgr.14176) 554 : cluster [DBG] pgmap v413: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:06.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:06.716 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:07.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:07 smithi006 bash[26022]: cluster 2023-11-17T05:46:05.946597+0000 mgr.smithi006.odexej (mgr.14176) 555 : cluster [DBG] pgmap v414: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:07.292 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:07.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:07 smithi156 bash[29655]: cluster 2023-11-17T05:46:05.946597+0000 mgr.smithi006.odexej (mgr.14176) 555 : cluster [DBG] pgmap v414: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:08.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:08.306 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:08 smithi006 bash[26022]: audit 2023-11-17T05:46:06.696851+0000 mgr.smithi006.odexej (mgr.14176) 556 : audit [DBG] from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:08 smithi156 bash[29655]: audit 2023-11-17T05:46:06.696851+0000 mgr.smithi006.odexej (mgr.14176) 556 : audit [DBG] from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:09.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:09 smithi006 bash[26022]: cluster 2023-11-17T05:46:07.947812+0000 mgr.smithi006.odexej (mgr.14176) 557 : cluster [DBG] pgmap v415: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:09 smithi156 bash[29655]: cluster 2023-11-17T05:46:07.947812+0000 mgr.smithi006.odexej (mgr.14176) 557 : cluster [DBG] pgmap v415: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:10.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:10 smithi006 bash[26022]: cluster 2023-11-17T05:46:09.949293+0000 mgr.smithi006.odexej (mgr.14176) 558 : cluster [DBG] pgmap v416: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:10 smithi156 bash[29655]: cluster 2023-11-17T05:46:09.949293+0000 mgr.smithi006.odexej (mgr.14176) 558 : cluster [DBG] pgmap v416: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:11.300 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:11.301 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:11.861 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:12.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:12 smithi006 bash[26022]: audit 2023-11-17T05:46:11.282764+0000 mgr.smithi006.odexej (mgr.14176) 559 : audit [DBG] from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:12.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:12 smithi006 bash[26022]: cluster 2023-11-17T05:46:11.950364+0000 mgr.smithi006.odexej (mgr.14176) 560 : cluster [DBG] pgmap v417: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:12.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:12 smithi156 bash[29655]: audit 2023-11-17T05:46:11.282764+0000 mgr.smithi006.odexej (mgr.14176) 559 : audit [DBG] from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:12.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:12 smithi156 bash[29655]: cluster 2023-11-17T05:46:11.950364+0000 mgr.smithi006.odexej (mgr.14176) 560 : cluster [DBG] pgmap v417: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:12.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:14.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:14 smithi006 bash[26022]: cluster 2023-11-17T05:46:13.951675+0000 mgr.smithi006.odexej (mgr.14176) 561 : cluster [DBG] pgmap v418: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:14.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:14 smithi156 bash[29655]: cluster 2023-11-17T05:46:13.951675+0000 mgr.smithi006.odexej (mgr.14176) 561 : cluster [DBG] pgmap v418: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:15.908 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:15.909 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:44:07.641389Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.ldakip on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:16.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:15 smithi006 bash[26022]: audit 2023-11-17T05:46:15.918851+0000 mon.smithi006 (mon.0) 761 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:46:16.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:15 smithi156 bash[29655]: audit 2023-11-17T05:46:15.918851+0000 mon.smithi006 (mon.0) 761 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:46:16.559 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:17.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:15.896251+0000 mgr.smithi006.odexej (mgr.14176) 562 : audit [DBG] from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:17.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: cluster 2023-11-17T05:46:15.952512+0000 mgr.smithi006.odexej (mgr.14176) 563 : cluster [DBG] pgmap v419: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:17.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:16.238292+0000 mon.smithi006 (mon.0) 762 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:46:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:16.240031+0000 mon.smithi006 (mon.0) 763 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:46:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:16.250646+0000 mon.smithi006 (mon.0) 764 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:16.259976+0000 mon.smithi006 (mon.0) 765 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:46:17.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:16 smithi006 bash[26022]: audit 2023-11-17T05:46:16.274005+0000 mon.smithi006 (mon.0) 766 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:17.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:15.896251+0000 mgr.smithi006.odexej (mgr.14176) 562 : audit [DBG] from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:17.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: cluster 2023-11-17T05:46:15.952512+0000 mgr.smithi006.odexej (mgr.14176) 563 : cluster [DBG] pgmap v419: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:16.238292+0000 mon.smithi006 (mon.0) 762 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:46:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:16.240031+0000 mon.smithi006 (mon.0) 763 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:46:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:16.250646+0000 mon.smithi006 (mon.0) 764 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:16.259976+0000 mon.smithi006 (mon.0) 765 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:46:17.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:16 smithi156 bash[29655]: audit 2023-11-17T05:46:16.274005+0000 mon.smithi006 (mon.0) 766 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:17.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:18.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:17 smithi006 bash[26022]: cluster 2023-11-17T05:46:16.251763+0000 mgr.smithi006.odexej (mgr.14176) 564 : cluster [DBG] pgmap v420: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:18.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:17 smithi006 bash[26022]: cephadm 2023-11-17T05:46:16.276868+0000 mgr.smithi006.odexej (mgr.14176) 565 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qxnmfc on smithi156 2023-11-17T05:46:18.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:17 smithi006 bash[26022]: cluster 2023-11-17T05:46:17.247166+0000 mon.smithi006 (mon.0) 767 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:46:18.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:17 smithi006 bash[26022]: cluster 2023-11-17T05:46:17.247234+0000 mon.smithi006 (mon.0) 768 : cluster [INF] Cluster is now healthy 2023-11-17T05:46:18.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:17 smithi156 bash[29655]: cluster 2023-11-17T05:46:16.251763+0000 mgr.smithi006.odexej (mgr.14176) 564 : cluster [DBG] pgmap v420: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:18.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:17 smithi156 bash[29655]: cephadm 2023-11-17T05:46:16.276868+0000 mgr.smithi006.odexej (mgr.14176) 565 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qxnmfc on smithi156 2023-11-17T05:46:18.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:17 smithi156 bash[29655]: cluster 2023-11-17T05:46:17.247166+0000 mon.smithi006 (mon.0) 767 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:46:18.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:17 smithi156 bash[29655]: cluster 2023-11-17T05:46:17.247234+0000 mon.smithi006 (mon.0) 768 : cluster [INF] Cluster is now healthy 2023-11-17T05:46:20.277 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:19 smithi006 bash[26022]: cluster 2023-11-17T05:46:18.252574+0000 mgr.smithi006.odexej (mgr.14176) 566 : cluster [DBG] pgmap v421: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:20.277 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:19 smithi006 bash[26022]: audit 2023-11-17T05:46:19.710124+0000 mon.smithi006 (mon.0) 769 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:20.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:19 smithi156 bash[29655]: cluster 2023-11-17T05:46:18.252574+0000 mgr.smithi006.odexej (mgr.14176) 566 : cluster [DBG] pgmap v421: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:20.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:19 smithi156 bash[29655]: audit 2023-11-17T05:46:19.710124+0000 mon.smithi006 (mon.0) 769 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:20.638 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:20.639 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:12.018253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dmllff on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dmllff\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dmllff\nDeploy daemon haproxy.nfs.foo.smithi156.dmllff ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.910677Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.wwldxr on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-wwldxr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.wwldxr\nDeploy daemon haproxy.nfs.foo.smithi006.wwldxr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:45:15.912867Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.hdaijn on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:21.229 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:21.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:20 smithi006 bash[26022]: audit 2023-11-17T05:46:20.046462+0000 mon.smithi006 (mon.0) 770 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qxnmfc"}]: dispatch 2023-11-17T05:46:21.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:21 smithi156 bash[29655]: audit 2023-11-17T05:46:20.046462+0000 mon.smithi006 (mon.0) 770 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qxnmfc"}]: dispatch 2023-11-17T05:46:22.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: cephadm 2023-11-17T05:46:20.045057+0000 mgr.smithi006.odexej (mgr.14176) 567 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: cephadm 2023-11-17T05:46:20.045670+0000 mgr.smithi006.odexej (mgr.14176) 568 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: cephadm 2023-11-17T05:46:20.047987+0000 mgr.smithi006.odexej (mgr.14176) 569 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: cephadm 2023-11-17T05:46:20.053980+0000 mgr.smithi006.odexej (mgr.14176) 570 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.rcebfv on smithi006 2023-11-17T05:46:22.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: cluster 2023-11-17T05:46:20.253357+0000 mgr.smithi006.odexej (mgr.14176) 571 : cluster [DBG] pgmap v422: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:22.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:22 smithi006 bash[26022]: audit 2023-11-17T05:46:20.617920+0000 mgr.smithi006.odexej (mgr.14176) 572 : audit [DBG] from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: cephadm 2023-11-17T05:46:20.045057+0000 mgr.smithi006.odexej (mgr.14176) 567 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:46:22.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: cephadm 2023-11-17T05:46:20.045670+0000 mgr.smithi006.odexej (mgr.14176) 568 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: cephadm 2023-11-17T05:46:20.047987+0000 mgr.smithi006.odexej (mgr.14176) 569 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qxnmfc ... 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: cephadm 2023-11-17T05:46:20.053980+0000 mgr.smithi006.odexej (mgr.14176) 570 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.rcebfv on smithi006 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: cluster 2023-11-17T05:46:20.253357+0000 mgr.smithi006.odexej (mgr.14176) 571 : cluster [DBG] pgmap v422: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:22.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:22 smithi156 bash[29655]: audit 2023-11-17T05:46:20.617920+0000 mgr.smithi006.odexej (mgr.14176) 572 : audit [DBG] from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:24.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:24 smithi006 bash[26022]: cluster 2023-11-17T05:46:22.254003+0000 mgr.smithi006.odexej (mgr.14176) 573 : cluster [DBG] pgmap v423: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:24.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:24 smithi156 bash[29655]: cluster 2023-11-17T05:46:22.254003+0000 mgr.smithi006.odexej (mgr.14176) 573 : cluster [DBG] pgmap v423: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:46:25.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:25 smithi006 bash[26022]: audit 2023-11-17T05:46:24.072579+0000 mon.smithi006 (mon.0) 771 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.rcebfv"}]: dispatch 2023-11-17T05:46:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:25 smithi156 bash[29655]: audit 2023-11-17T05:46:24.072579+0000 mon.smithi006 (mon.0) 771 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.rcebfv"}]: dispatch 2023-11-17T05:46:25.520 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:25.520 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:26.145 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:26.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cephadm 2023-11-17T05:46:24.071849+0000 mgr.smithi006.odexej (mgr.14176) 574 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cephadm 2023-11-17T05:46:24.072091+0000 mgr.smithi006.odexej (mgr.14176) 575 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cephadm 2023-11-17T05:46:24.073692+0000 mgr.smithi006.odexej (mgr.14176) 576 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:46:26.423 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cephadm 2023-11-17T05:46:24.075579+0000 mgr.smithi006.odexej (mgr.14176) 577 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cephadm 2023-11-17T05:46:24.077270+0000 mgr.smithi006.odexej (mgr.14176) 578 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cluster 2023-11-17T05:46:24.078580+0000 mgr.smithi006.odexej (mgr.14176) 579 : cluster [DBG] pgmap v424: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: cluster 2023-11-17T05:46:25.006458+0000 mon.smithi006 (mon.0) 772 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: audit 2023-11-17T05:46:25.294580+0000 mon.smithi006 (mon.0) 773 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:26.424 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:26 smithi006 bash[26022]: audit 2023-11-17T05:46:25.501269+0000 mgr.smithi006.odexej (mgr.14176) 580 : audit [DBG] from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:26.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cephadm 2023-11-17T05:46:24.071849+0000 mgr.smithi006.odexej (mgr.14176) 574 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:46:26.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:46:26.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:46:26.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:46:26.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:46:26.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.443 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.443 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cephadm 2023-11-17T05:46:24.072091+0000 mgr.smithi006.odexej (mgr.14176) 575 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.443 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cephadm 2023-11-17T05:46:24.073692+0000 mgr.smithi006.odexej (mgr.14176) 576 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.443 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rcebfv ... 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cephadm 2023-11-17T05:46:24.075579+0000 mgr.smithi006.odexej (mgr.14176) 577 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cephadm 2023-11-17T05:46:24.077270+0000 mgr.smithi006.odexej (mgr.14176) 578 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cluster 2023-11-17T05:46:24.078580+0000 mgr.smithi006.odexej (mgr.14176) 579 : cluster [DBG] pgmap v424: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:46:26.445 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: cluster 2023-11-17T05:46:25.006458+0000 mon.smithi006 (mon.0) 772 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:46:26.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: audit 2023-11-17T05:46:25.294580+0000 mon.smithi006 (mon.0) 773 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:26.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:26 smithi156 bash[29655]: audit 2023-11-17T05:46:25.501269+0000 mgr.smithi006.odexej (mgr.14176) 580 : audit [DBG] from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:27.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:27.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:27 smithi006 bash[26022]: cluster 2023-11-17T05:46:26.079956+0000 mgr.smithi006.odexej (mgr.14176) 581 : cluster [DBG] pgmap v425: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:46:27.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:27 smithi156 bash[29655]: cluster 2023-11-17T05:46:26.079956+0000 mgr.smithi006.odexej (mgr.14176) 581 : cluster [DBG] pgmap v425: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:46:29.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:29 smithi006 bash[26022]: cluster 2023-11-17T05:46:28.081050+0000 mgr.smithi006.odexej (mgr.14176) 582 : cluster [DBG] pgmap v426: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:29.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:29 smithi156 bash[29655]: cluster 2023-11-17T05:46:28.081050+0000 mgr.smithi006.odexej (mgr.14176) 582 : cluster [DBG] pgmap v426: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:30.212 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:30.212 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:30.848 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:31.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:31 smithi006 bash[26022]: cluster 2023-11-17T05:46:30.082411+0000 mgr.smithi006.odexej (mgr.14176) 583 : cluster [DBG] pgmap v427: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:31.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:31 smithi006 bash[26022]: audit 2023-11-17T05:46:30.193070+0000 mgr.smithi006.odexej (mgr.14176) 584 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:31.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:31 smithi156 bash[29655]: cluster 2023-11-17T05:46:30.082411+0000 mgr.smithi006.odexej (mgr.14176) 583 : cluster [DBG] pgmap v427: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:31.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:31 smithi156 bash[29655]: audit 2023-11-17T05:46:30.193070+0000 mgr.smithi006.odexej (mgr.14176) 584 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:31.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:33.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:33 smithi006 bash[26022]: cluster 2023-11-17T05:46:32.083177+0000 mgr.smithi006.odexej (mgr.14176) 585 : cluster [DBG] pgmap v428: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:33.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:33 smithi156 bash[29655]: cluster 2023-11-17T05:46:32.083177+0000 mgr.smithi006.odexej (mgr.14176) 585 : cluster [DBG] pgmap v428: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T05:46:35.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:35.013 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:35.637 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:35.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:35 smithi006 bash[26022]: cluster 2023-11-17T05:46:34.084368+0000 mgr.smithi006.odexej (mgr.14176) 586 : cluster [DBG] pgmap v429: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T05:46:35.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:35 smithi006 bash[26022]: audit 2023-11-17T05:46:34.707228+0000 mon.smithi006 (mon.0) 774 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:35.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:35 smithi006 bash[26022]: audit 2023-11-17T05:46:34.993581+0000 mgr.smithi006.odexej (mgr.14176) 587 : audit [DBG] from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:36.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:35 smithi156 bash[29655]: cluster 2023-11-17T05:46:34.084368+0000 mgr.smithi006.odexej (mgr.14176) 586 : cluster [DBG] pgmap v429: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T05:46:36.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:35 smithi156 bash[29655]: audit 2023-11-17T05:46:34.707228+0000 mon.smithi006 (mon.0) 774 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:46:36.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:35 smithi156 bash[29655]: audit 2023-11-17T05:46:34.993581+0000 mgr.smithi006.odexej (mgr.14176) 587 : audit [DBG] from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:36.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:37.949 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:37 smithi006 bash[26022]: cluster 2023-11-17T05:46:36.085717+0000 mgr.smithi006.odexej (mgr.14176) 588 : cluster [DBG] pgmap v430: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:38.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:37 smithi156 bash[29655]: cluster 2023-11-17T05:46:36.085717+0000 mgr.smithi006.odexej (mgr.14176) 588 : cluster [DBG] pgmap v430: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:39.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:40.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:39 smithi006 bash[26022]: cluster 2023-11-17T05:46:38.086788+0000 mgr.smithi006.odexej (mgr.14176) 589 : cluster [DBG] pgmap v431: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:39 smithi156 bash[29655]: cluster 2023-11-17T05:46:38.086788+0000 mgr.smithi006.odexej (mgr.14176) 589 : cluster [DBG] pgmap v431: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:40.288 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:41.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:40 smithi006 bash[26022]: audit 2023-11-17T05:46:39.675279+0000 mgr.smithi006.odexej (mgr.14176) 590 : audit [DBG] from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:41.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:40 smithi156 bash[29655]: audit 2023-11-17T05:46:39.675279+0000 mgr.smithi006.odexej (mgr.14176) 590 : audit [DBG] from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:41.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:42.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:41 smithi006 bash[26022]: cluster 2023-11-17T05:46:40.087739+0000 mgr.smithi006.odexej (mgr.14176) 591 : cluster [DBG] pgmap v432: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:42.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:41 smithi156 bash[29655]: cluster 2023-11-17T05:46:40.087739+0000 mgr.smithi006.odexej (mgr.14176) 591 : cluster [DBG] pgmap v432: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:43.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:43 smithi006 bash[26022]: cluster 2023-11-17T05:46:42.088823+0000 mgr.smithi006.odexej (mgr.14176) 592 : cluster [DBG] pgmap v433: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:43.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:43 smithi156 bash[29655]: cluster 2023-11-17T05:46:42.088823+0000 mgr.smithi006.odexej (mgr.14176) 592 : cluster [DBG] pgmap v433: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:44.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:44.500 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:45.077 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:46.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:46.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:45 smithi006 bash[26022]: cluster 2023-11-17T05:46:44.090314+0000 mgr.smithi006.odexej (mgr.14176) 593 : cluster [DBG] pgmap v434: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:46.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:45 smithi006 bash[26022]: audit 2023-11-17T05:46:44.482802+0000 mgr.smithi006.odexej (mgr.14176) 594 : audit [DBG] from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:46.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:45 smithi156 bash[29655]: cluster 2023-11-17T05:46:44.090314+0000 mgr.smithi006.odexej (mgr.14176) 593 : cluster [DBG] pgmap v434: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:46.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:45 smithi156 bash[29655]: audit 2023-11-17T05:46:44.482802+0000 mgr.smithi006.odexej (mgr.14176) 594 : audit [DBG] from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:48.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:47 smithi006 bash[26022]: cluster 2023-11-17T05:46:46.091807+0000 mgr.smithi006.odexej (mgr.14176) 595 : cluster [DBG] pgmap v435: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:48.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:47 smithi156 bash[29655]: cluster 2023-11-17T05:46:46.091807+0000 mgr.smithi006.odexej (mgr.14176) 595 : cluster [DBG] pgmap v435: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:49.140 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:49.140 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:49.757 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:50.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:49 smithi006 bash[26022]: cluster 2023-11-17T05:46:48.092865+0000 mgr.smithi006.odexej (mgr.14176) 596 : cluster [DBG] pgmap v436: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:50.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:49 smithi156 bash[29655]: cluster 2023-11-17T05:46:48.092865+0000 mgr.smithi006.odexej (mgr.14176) 596 : cluster [DBG] pgmap v436: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:50.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:51.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:50 smithi006 bash[26022]: audit 2023-11-17T05:46:49.123458+0000 mgr.smithi006.odexej (mgr.14176) 597 : audit [DBG] from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:51.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:50 smithi156 bash[29655]: audit 2023-11-17T05:46:49.123458+0000 mgr.smithi006.odexej (mgr.14176) 597 : audit [DBG] from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:52.044 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:51 smithi006 bash[26022]: cluster 2023-11-17T05:46:50.094227+0000 mgr.smithi006.odexej (mgr.14176) 598 : cluster [DBG] pgmap v437: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:52.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:51 smithi156 bash[29655]: cluster 2023-11-17T05:46:50.094227+0000 mgr.smithi006.odexej (mgr.14176) 598 : cluster [DBG] pgmap v437: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:53.766 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:53.766 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:53.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:53 smithi156 bash[29655]: cluster 2023-11-17T05:46:52.095349+0000 mgr.smithi006.odexej (mgr.14176) 599 : cluster [DBG] pgmap v438: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:54.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:53 smithi006 bash[26022]: cluster 2023-11-17T05:46:52.095349+0000 mgr.smithi006.odexej (mgr.14176) 599 : cluster [DBG] pgmap v438: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:54.379 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:46:55.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:54 smithi006 bash[26022]: audit 2023-11-17T05:46:53.746440+0000 mgr.smithi006.odexej (mgr.14176) 600 : audit [DBG] from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:55.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:54 smithi156 bash[29655]: audit 2023-11-17T05:46:53.746440+0000 mgr.smithi006.odexej (mgr.14176) 600 : audit [DBG] from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:46:55.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:46:56.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:55 smithi006 bash[26022]: cluster 2023-11-17T05:46:54.096570+0000 mgr.smithi006.odexej (mgr.14176) 601 : cluster [DBG] pgmap v439: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:56.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:55 smithi156 bash[29655]: cluster 2023-11-17T05:46:54.096570+0000 mgr.smithi006.odexej (mgr.14176) 601 : cluster [DBG] pgmap v439: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:46:58.110 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:57 smithi006 bash[26022]: cluster 2023-11-17T05:46:56.098052+0000 mgr.smithi006.odexej (mgr.14176) 602 : cluster [DBG] pgmap v440: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:58.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:57 smithi156 bash[29655]: cluster 2023-11-17T05:46:56.098052+0000 mgr.smithi006.odexej (mgr.14176) 602 : cluster [DBG] pgmap v440: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:46:58.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:46:58.475 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:46:59.037 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:47:00.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:47:00.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:59 smithi006 bash[26022]: cluster 2023-11-17T05:46:58.099275+0000 mgr.smithi006.odexej (mgr.14176) 603 : cluster [DBG] pgmap v441: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:47:00.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:46:59 smithi006 bash[26022]: audit 2023-11-17T05:46:58.456204+0000 mgr.smithi006.odexej (mgr.14176) 604 : audit [DBG] from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:47:00.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:59 smithi156 bash[29655]: cluster 2023-11-17T05:46:58.099275+0000 mgr.smithi006.odexej (mgr.14176) 603 : cluster [DBG] pgmap v441: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:47:00.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:46:59 smithi156 bash[29655]: audit 2023-11-17T05:46:58.456204+0000 mgr.smithi006.odexej (mgr.14176) 604 : audit [DBG] from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:47:02.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:01 smithi006 bash[26022]: cluster 2023-11-17T05:47:00.100757+0000 mgr.smithi006.odexej (mgr.14176) 605 : cluster [DBG] pgmap v442: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:47:02.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:01 smithi156 bash[29655]: cluster 2023-11-17T05:47:00.100757+0000 mgr.smithi006.odexej (mgr.14176) 605 : cluster [DBG] pgmap v442: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:47:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:01.897 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:45:15.914721Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tkrtfh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:20.047801Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qxnmfc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qxnmfc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qxnmfc\nDeploy daemon haproxy.nfs.foo.smithi156.qxnmfc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.073573Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rcebfv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rcebfv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rcebfv\nDeploy daemon haproxy.nfs.foo.smithi006.rcebfv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:46:24.075451Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.yywdxw on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:01.904 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:03 smithi156 bash[29655]: cluster 2023-11-17T05:47:02.101832+0000 mgr.smithi006.odexej (mgr.14176) 606 : cluster [DBG] pgmap v443: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:04 smithi156 bash[29655]: audit 2023-11-17T05:47:03.208230+0000 mgr.smithi006.odexej (mgr.14176) 607 : audit [DBG] from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:05 smithi156 bash[29655]: cluster 2023-11-17T05:47:04.103021+0000 mgr.smithi006.odexej (mgr.14176) 608 : cluster [DBG] pgmap v444: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:07 smithi156 bash[29655]: cluster 2023-11-17T05:47:06.104421+0000 mgr.smithi006.odexej (mgr.14176) 609 : cluster [DBG] pgmap v445: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:09 smithi156 bash[29655]: cluster 2023-11-17T05:47:08.105455+0000 mgr.smithi006.odexej (mgr.14176) 610 : cluster [DBG] pgmap v446: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:11 smithi156 bash[29655]: cluster 2023-11-17T05:47:10.106896+0000 mgr.smithi006.odexej (mgr.14176) 611 : cluster [DBG] pgmap v447: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:13 smithi156 bash[29655]: cluster 2023-11-17T05:47:12.108032+0000 mgr.smithi006.odexej (mgr.14176) 612 : cluster [DBG] pgmap v448: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.910 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:15 smithi156 bash[29655]: cluster 2023-11-17T05:47:14.109203+0000 mgr.smithi006.odexej (mgr.14176) 613 : cluster [DBG] pgmap v449: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:01.911 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:17 smithi156 bash[29655]: cluster 2023-11-17T05:47:16.110729+0000 mgr.smithi006.odexej (mgr.14176) 614 : cluster [DBG] pgmap v450: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.911 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:19 smithi156 bash[29655]: cluster 2023-11-17T05:47:18.111939+0000 mgr.smithi006.odexej (mgr.14176) 615 : cluster [DBG] pgmap v451: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.962 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:21 smithi156 bash[29655]: cluster 2023-11-17T05:47:20.113405+0000 mgr.smithi006.odexej (mgr.14176) 616 : cluster [DBG] pgmap v452: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.962 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:23 smithi156 bash[29655]: cluster 2023-11-17T05:47:22.114453+0000 mgr.smithi006.odexej (mgr.14176) 617 : cluster [DBG] pgmap v453: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:01.962 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.081109+0000 mon.smithi006 (mon.0) 775 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.402211+0000 mon.smithi006 (mon.0) 776 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.404041+0000 mon.smithi006 (mon.0) 777 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.405773+0000 mon.smithi006 (mon.0) 778 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.407396+0000 mon.smithi006 (mon.0) 779 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.413760+0000 mon.smithi006 (mon.0) 780 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.420084+0000 mon.smithi006 (mon.0) 781 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.421897+0000 mon.smithi006 (mon.0) 782 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.423163+0000 mon.smithi006 (mon.0) 783 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.139 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.424431+0000 mon.smithi006 (mon.0) 784 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.432111+0000 mon.smithi006 (mon.0) 785 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.433942+0000 mon.smithi006 (mon.0) 786 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.435067+0000 mon.smithi006 (mon.0) 787 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.441622+0000 mon.smithi006 (mon.0) 788 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.450942+0000 mon.smithi006 (mon.0) 789 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:24 smithi156 bash[29655]: audit 2023-11-17T05:47:24.463159+0000 mon.smithi006 (mon.0) 790 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cluster 2023-11-17T05:47:24.115473+0000 mgr.smithi006.odexej (mgr.14176) 618 : cluster [DBG] pgmap v454: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cephadm 2023-11-17T05:47:24.408560+0000 mgr.smithi006.odexej (mgr.14176) 619 : cephadm [INF] Adjusting osd_memory_target on smithi156 to 3017M 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cephadm 2023-11-17T05:47:24.425426+0000 mgr.smithi006.odexej (mgr.14176) 620 : cephadm [INF] Adjusting osd_memory_target on smithi006 to 1994M 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cluster 2023-11-17T05:47:24.442578+0000 mgr.smithi006.odexej (mgr.14176) 621 : cluster [DBG] pgmap v455: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cephadm 2023-11-17T05:47:24.466064+0000 mgr.smithi006.odexej (mgr.14176) 622 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.gmbhov on smithi156 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cluster 2023-11-17T05:47:24.883087+0000 mon.smithi006 (mon.0) 791 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:48:02.140 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:25 smithi156 bash[29655]: cluster 2023-11-17T05:47:24.883153+0000 mon.smithi006 (mon.0) 792 : cluster [INF] Cluster is now healthy 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:27 smithi156 bash[29655]: cluster 2023-11-17T05:47:26.443300+0000 mgr.smithi006.odexej (mgr.14176) 623 : cluster [DBG] pgmap v456: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:28 smithi156 bash[29655]: audit 2023-11-17T05:47:28.175975+0000 mon.smithi006 (mon.0) 793 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.gmbhov"}]: dispatch 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: cephadm 2023-11-17T05:47:28.175024+0000 mgr.smithi006.odexej (mgr.14176) 624 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.141 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.142 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.142 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:48:02.142 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:02.142 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:48:02.142 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.143 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: cephadm 2023-11-17T05:47:28.175383+0000 mgr.smithi006.odexej (mgr.14176) 625 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: cephadm 2023-11-17T05:47:28.177204+0000 mgr.smithi006.odexej (mgr.14176) 626 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.144 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: cephadm 2023-11-17T05:47:28.179781+0000 mgr.smithi006.odexej (mgr.14176) 627 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.sfefkl on smithi006 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:29 smithi156 bash[29655]: cluster 2023-11-17T05:47:28.444053+0000 mgr.smithi006.odexej (mgr.14176) 628 : cluster [DBG] pgmap v457: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:31 smithi156 bash[29655]: cluster 2023-11-17T05:47:30.445238+0000 mgr.smithi006.odexej (mgr.14176) 629 : cluster [DBG] pgmap v458: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cephadm 2023-11-17T05:47:31.938868+0000 mgr.smithi006.odexej (mgr.14176) 630 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.145 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.146 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cephadm 2023-11-17T05:47:31.939113+0000 mgr.smithi006.odexej (mgr.14176) 631 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: audit 2023-11-17T05:47:31.939647+0000 mon.smithi006 (mon.0) 794 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.sfefkl"}]: dispatch 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cephadm 2023-11-17T05:47:31.940843+0000 mgr.smithi006.odexej (mgr.14176) 632 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cephadm 2023-11-17T05:47:31.942740+0000 mgr.smithi006.odexej (mgr.14176) 633 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cephadm 2023-11-17T05:47:31.944422+0000 mgr.smithi006.odexej (mgr.14176) 634 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:32 smithi156 bash[29655]: cluster 2023-11-17T05:47:31.945365+0000 mgr.smithi006.odexej (mgr.14176) 635 : cluster [DBG] pgmap v459: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:33 smithi156 bash[29655]: cluster 2023-11-17T05:47:32.503000+0000 mon.smithi006 (mon.0) 795 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:34 smithi156 bash[29655]: cluster 2023-11-17T05:47:33.946513+0000 mgr.smithi006.odexej (mgr.14176) 636 : cluster [DBG] pgmap v460: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:48:02.148 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:36 smithi156 bash[29655]: audit 2023-11-17T05:47:35.348187+0000 mon.smithi006 (mon.0) 796 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:36 smithi156 bash[29655]: cluster 2023-11-17T05:47:35.947879+0000 mgr.smithi006.odexej (mgr.14176) 637 : cluster [DBG] pgmap v461: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:39 smithi156 bash[29655]: cluster 2023-11-17T05:47:37.948879+0000 mgr.smithi006.odexej (mgr.14176) 638 : cluster [DBG] pgmap v462: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:41 smithi156 bash[29655]: cluster 2023-11-17T05:47:39.950236+0000 mgr.smithi006.odexej (mgr.14176) 639 : cluster [DBG] pgmap v463: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:42 smithi156 bash[29655]: cluster 2023-11-17T05:47:41.951286+0000 mgr.smithi006.odexej (mgr.14176) 640 : cluster [DBG] pgmap v464: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:45 smithi156 bash[29655]: cluster 2023-11-17T05:47:43.952455+0000 mgr.smithi006.odexej (mgr.14176) 641 : cluster [DBG] pgmap v465: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:47 smithi156 bash[29655]: cluster 2023-11-17T05:47:45.953794+0000 mgr.smithi006.odexej (mgr.14176) 642 : cluster [DBG] pgmap v466: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:49 smithi156 bash[29655]: cluster 2023-11-17T05:47:47.954820+0000 mgr.smithi006.odexej (mgr.14176) 643 : cluster [DBG] pgmap v467: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:51 smithi156 bash[29655]: cluster 2023-11-17T05:47:49.956217+0000 mgr.smithi006.odexej (mgr.14176) 644 : cluster [DBG] pgmap v468: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:52 smithi156 bash[29655]: cluster 2023-11-17T05:47:51.957226+0000 mgr.smithi006.odexej (mgr.14176) 645 : cluster [DBG] pgmap v469: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:55 smithi156 bash[29655]: cluster 2023-11-17T05:47:53.958389+0000 mgr.smithi006.odexej (mgr.14176) 646 : cluster [DBG] pgmap v470: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:57 smithi156 bash[29655]: cluster 2023-11-17T05:47:55.959755+0000 mgr.smithi006.odexej (mgr.14176) 647 : cluster [DBG] pgmap v471: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:47:59 smithi156 bash[29655]: cluster 2023-11-17T05:47:57.960893+0000 mgr.smithi006.odexej (mgr.14176) 648 : cluster [DBG] pgmap v472: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.149 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:00 smithi156 bash[29655]: cluster 2023-11-17T05:47:59.962319+0000 mgr.smithi006.odexej (mgr.14176) 649 : cluster [DBG] pgmap v473: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.150 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:03 smithi006 bash[26022]: cluster 2023-11-17T05:47:02.101832+0000 mgr.smithi006.odexej (mgr.14176) 606 : cluster [DBG] pgmap v443: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.150 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:04 smithi006 bash[26022]: audit 2023-11-17T05:47:03.208230+0000 mgr.smithi006.odexej (mgr.14176) 607 : audit [DBG] from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:02.150 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:05 smithi006 bash[26022]: cluster 2023-11-17T05:47:04.103021+0000 mgr.smithi006.odexej (mgr.14176) 608 : cluster [DBG] pgmap v444: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:07 smithi006 bash[26022]: cluster 2023-11-17T05:47:06.104421+0000 mgr.smithi006.odexej (mgr.14176) 609 : cluster [DBG] pgmap v445: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:09 smithi006 bash[26022]: cluster 2023-11-17T05:47:08.105455+0000 mgr.smithi006.odexej (mgr.14176) 610 : cluster [DBG] pgmap v446: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:11 smithi006 bash[26022]: cluster 2023-11-17T05:47:10.106896+0000 mgr.smithi006.odexej (mgr.14176) 611 : cluster [DBG] pgmap v447: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:13 smithi006 bash[26022]: cluster 2023-11-17T05:47:12.108032+0000 mgr.smithi006.odexej (mgr.14176) 612 : cluster [DBG] pgmap v448: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:15 smithi006 bash[26022]: cluster 2023-11-17T05:47:14.109203+0000 mgr.smithi006.odexej (mgr.14176) 613 : cluster [DBG] pgmap v449: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:17 smithi006 bash[26022]: cluster 2023-11-17T05:47:16.110729+0000 mgr.smithi006.odexej (mgr.14176) 614 : cluster [DBG] pgmap v450: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:19 smithi006 bash[26022]: cluster 2023-11-17T05:47:18.111939+0000 mgr.smithi006.odexej (mgr.14176) 615 : cluster [DBG] pgmap v451: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:21 smithi006 bash[26022]: cluster 2023-11-17T05:47:20.113405+0000 mgr.smithi006.odexej (mgr.14176) 616 : cluster [DBG] pgmap v452: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:23 smithi006 bash[26022]: cluster 2023-11-17T05:47:22.114453+0000 mgr.smithi006.odexej (mgr.14176) 617 : cluster [DBG] pgmap v453: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.081109+0000 mon.smithi006 (mon.0) 775 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.402211+0000 mon.smithi006 (mon.0) 776 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.404041+0000 mon.smithi006 (mon.0) 777 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.151 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.405773+0000 mon.smithi006 (mon.0) 778 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.407396+0000 mon.smithi006 (mon.0) 779 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.413760+0000 mon.smithi006 (mon.0) 780 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.420084+0000 mon.smithi006 (mon.0) 781 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.421897+0000 mon.smithi006 (mon.0) 782 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.423163+0000 mon.smithi006 (mon.0) 783 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.424431+0000 mon.smithi006 (mon.0) 784 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.432111+0000 mon.smithi006 (mon.0) 785 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.433942+0000 mon.smithi006 (mon.0) 786 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.435067+0000 mon.smithi006 (mon.0) 787 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.441622+0000 mon.smithi006 (mon.0) 788 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.450942+0000 mon.smithi006 (mon.0) 789 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:24 smithi006 bash[26022]: audit 2023-11-17T05:47:24.463159+0000 mon.smithi006 (mon.0) 790 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cluster 2023-11-17T05:47:24.115473+0000 mgr.smithi006.odexej (mgr.14176) 618 : cluster [DBG] pgmap v454: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cephadm 2023-11-17T05:47:24.408560+0000 mgr.smithi006.odexej (mgr.14176) 619 : cephadm [INF] Adjusting osd_memory_target on smithi156 to 3017M 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cephadm 2023-11-17T05:47:24.425426+0000 mgr.smithi006.odexej (mgr.14176) 620 : cephadm [INF] Adjusting osd_memory_target on smithi006 to 1994M 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cluster 2023-11-17T05:47:24.442578+0000 mgr.smithi006.odexej (mgr.14176) 621 : cluster [DBG] pgmap v455: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cephadm 2023-11-17T05:47:24.466064+0000 mgr.smithi006.odexej (mgr.14176) 622 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.gmbhov on smithi156 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cluster 2023-11-17T05:47:24.883087+0000 mon.smithi006 (mon.0) 791 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:25 smithi006 bash[26022]: cluster 2023-11-17T05:47:24.883153+0000 mon.smithi006 (mon.0) 792 : cluster [INF] Cluster is now healthy 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:27 smithi006 bash[26022]: cluster 2023-11-17T05:47:26.443300+0000 mgr.smithi006.odexej (mgr.14176) 623 : cluster [DBG] pgmap v456: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:28 smithi006 bash[26022]: audit 2023-11-17T05:47:28.175975+0000 mon.smithi006 (mon.0) 793 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.gmbhov"}]: dispatch 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: cephadm 2023-11-17T05:47:28.175024+0000 mgr.smithi006.odexej (mgr.14176) 624 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.153 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: cephadm 2023-11-17T05:47:28.175383+0000 mgr.smithi006.odexej (mgr.14176) 625 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: cephadm 2023-11-17T05:47:28.177204+0000 mgr.smithi006.odexej (mgr.14176) 626 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov 2023-11-17T05:48:02.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gmbhov ... 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: cephadm 2023-11-17T05:47:28.179781+0000 mgr.smithi006.odexej (mgr.14176) 627 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.sfefkl on smithi006 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:29 smithi006 bash[26022]: cluster 2023-11-17T05:47:28.444053+0000 mgr.smithi006.odexej (mgr.14176) 628 : cluster [DBG] pgmap v457: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:31 smithi006 bash[26022]: cluster 2023-11-17T05:47:30.445238+0000 mgr.smithi006.odexej (mgr.14176) 629 : cluster [DBG] pgmap v458: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cephadm 2023-11-17T05:47:31.938868+0000 mgr.smithi006.odexej (mgr.14176) 630 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cephadm 2023-11-17T05:47:31.939113+0000 mgr.smithi006.odexej (mgr.14176) 631 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: audit 2023-11-17T05:47:31.939647+0000 mon.smithi006 (mon.0) 794 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.sfefkl"}]: dispatch 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cephadm 2023-11-17T05:47:31.940843+0000 mgr.smithi006.odexej (mgr.14176) 632 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sfefkl ... 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cephadm 2023-11-17T05:47:31.942740+0000 mgr.smithi006.odexej (mgr.14176) 633 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:02.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cephadm 2023-11-17T05:47:31.944422+0000 mgr.smithi006.odexej (mgr.14176) 634 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:32 smithi006 bash[26022]: cluster 2023-11-17T05:47:31.945365+0000 mgr.smithi006.odexej (mgr.14176) 635 : cluster [DBG] pgmap v459: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:33 smithi006 bash[26022]: cluster 2023-11-17T05:47:32.503000+0000 mon.smithi006 (mon.0) 795 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:34 smithi006 bash[26022]: cluster 2023-11-17T05:47:33.946513+0000 mgr.smithi006.odexej (mgr.14176) 636 : cluster [DBG] pgmap v460: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:36 smithi006 bash[26022]: audit 2023-11-17T05:47:35.348187+0000 mon.smithi006 (mon.0) 796 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:36 smithi006 bash[26022]: cluster 2023-11-17T05:47:35.947879+0000 mgr.smithi006.odexej (mgr.14176) 637 : cluster [DBG] pgmap v461: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:39 smithi006 bash[26022]: cluster 2023-11-17T05:47:37.948879+0000 mgr.smithi006.odexej (mgr.14176) 638 : cluster [DBG] pgmap v462: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:41 smithi006 bash[26022]: cluster 2023-11-17T05:47:39.950236+0000 mgr.smithi006.odexej (mgr.14176) 639 : cluster [DBG] pgmap v463: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:42 smithi006 bash[26022]: cluster 2023-11-17T05:47:41.951286+0000 mgr.smithi006.odexej (mgr.14176) 640 : cluster [DBG] pgmap v464: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:45 smithi006 bash[26022]: cluster 2023-11-17T05:47:43.952455+0000 mgr.smithi006.odexej (mgr.14176) 641 : cluster [DBG] pgmap v465: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:47 smithi006 bash[26022]: cluster 2023-11-17T05:47:45.953794+0000 mgr.smithi006.odexej (mgr.14176) 642 : cluster [DBG] pgmap v466: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:49 smithi006 bash[26022]: cluster 2023-11-17T05:47:47.954820+0000 mgr.smithi006.odexej (mgr.14176) 643 : cluster [DBG] pgmap v467: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:51 smithi006 bash[26022]: cluster 2023-11-17T05:47:49.956217+0000 mgr.smithi006.odexej (mgr.14176) 644 : cluster [DBG] pgmap v468: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:52 smithi006 bash[26022]: cluster 2023-11-17T05:47:51.957226+0000 mgr.smithi006.odexej (mgr.14176) 645 : cluster [DBG] pgmap v469: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:55 smithi006 bash[26022]: cluster 2023-11-17T05:47:53.958389+0000 mgr.smithi006.odexej (mgr.14176) 646 : cluster [DBG] pgmap v470: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:02.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:57 smithi006 bash[26022]: cluster 2023-11-17T05:47:55.959755+0000 mgr.smithi006.odexej (mgr.14176) 647 : cluster [DBG] pgmap v471: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:47:59 smithi006 bash[26022]: cluster 2023-11-17T05:47:57.960893+0000 mgr.smithi006.odexej (mgr.14176) 648 : cluster [DBG] pgmap v472: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:00 smithi006 bash[26022]: cluster 2023-11-17T05:47:59.962319+0000 mgr.smithi006.odexej (mgr.14176) 649 : cluster [DBG] pgmap v473: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:02 smithi006 bash[26022]: cluster 2023-11-17T05:48:01.963337+0000 mgr.smithi006.odexej (mgr.14176) 650 : cluster [DBG] pgmap v474: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:02 smithi156 bash[29655]: cluster 2023-11-17T05:48:01.963337+0000 mgr.smithi006.odexej (mgr.14176) 650 : cluster [DBG] pgmap v474: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:02.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:05.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:05 smithi006 bash[26022]: cluster 2023-11-17T05:48:03.964527+0000 mgr.smithi006.odexej (mgr.14176) 651 : cluster [DBG] pgmap v475: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:05.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:05 smithi156 bash[29655]: cluster 2023-11-17T05:48:03.964527+0000 mgr.smithi006.odexej (mgr.14176) 651 : cluster [DBG] pgmap v475: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:06.143 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:06.143 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:06.685 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:07.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:07 smithi006 bash[26022]: cluster 2023-11-17T05:48:05.965686+0000 mgr.smithi006.odexej (mgr.14176) 652 : cluster [DBG] pgmap v476: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:07.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:07 smithi156 bash[29655]: cluster 2023-11-17T05:48:05.965686+0000 mgr.smithi006.odexej (mgr.14176) 652 : cluster [DBG] pgmap v476: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:07.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:08.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:08 smithi006 bash[26022]: audit 2023-11-17T05:48:06.131334+0000 mgr.smithi006.odexej (mgr.14176) 653 : audit [DBG] from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:08.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:08 smithi156 bash[29655]: audit 2023-11-17T05:48:06.131334+0000 mgr.smithi006.odexej (mgr.14176) 653 : audit [DBG] from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:09.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:09 smithi006 bash[26022]: cluster 2023-11-17T05:48:07.966757+0000 mgr.smithi006.odexej (mgr.14176) 654 : cluster [DBG] pgmap v477: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:09 smithi156 bash[29655]: cluster 2023-11-17T05:48:07.966757+0000 mgr.smithi006.odexej (mgr.14176) 654 : cluster [DBG] pgmap v477: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:10.358 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:10 smithi006 bash[26022]: cluster 2023-11-17T05:48:09.968202+0000 mgr.smithi006.odexej (mgr.14176) 655 : cluster [DBG] pgmap v478: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:10.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:10 smithi156 bash[29655]: cluster 2023-11-17T05:48:09.968202+0000 mgr.smithi006.odexej (mgr.14176) 655 : cluster [DBG] pgmap v478: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:10.724 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:10.724 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:11.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:11 smithi006 bash[26022]: audit 2023-11-17T05:48:10.704457+0000 mgr.smithi006.odexej (mgr.14176) 656 : audit [DBG] from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:11.354 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:11.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:11 smithi156 bash[29655]: audit 2023-11-17T05:48:10.704457+0000 mgr.smithi006.odexej (mgr.14176) 656 : audit [DBG] from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:12.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:12.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:12 smithi006 bash[26022]: cluster 2023-11-17T05:48:11.969214+0000 mgr.smithi006.odexej (mgr.14176) 657 : cluster [DBG] pgmap v479: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:12.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:12 smithi156 bash[29655]: cluster 2023-11-17T05:48:11.969214+0000 mgr.smithi006.odexej (mgr.14176) 657 : cluster [DBG] pgmap v479: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:15.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:15 smithi006 bash[26022]: cluster 2023-11-17T05:48:13.970434+0000 mgr.smithi006.odexej (mgr.14176) 658 : cluster [DBG] pgmap v480: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:15.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:15 smithi156 bash[29655]: cluster 2023-11-17T05:48:13.970434+0000 mgr.smithi006.odexej (mgr.14176) 658 : cluster [DBG] pgmap v480: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:15.473 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:15.474 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:16.061 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:16.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:16 smithi006 bash[26022]: audit 2023-11-17T05:48:15.454082+0000 mgr.smithi006.odexej (mgr.14176) 659 : audit [DBG] from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:16.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:16 smithi006 bash[26022]: cluster 2023-11-17T05:48:15.971889+0000 mgr.smithi006.odexej (mgr.14176) 660 : cluster [DBG] pgmap v481: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:16.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:16 smithi156 bash[29655]: audit 2023-11-17T05:48:15.454082+0000 mgr.smithi006.odexej (mgr.14176) 659 : audit [DBG] from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:16.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:16 smithi156 bash[29655]: cluster 2023-11-17T05:48:15.971889+0000 mgr.smithi006.odexej (mgr.14176) 660 : cluster [DBG] pgmap v481: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:17.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:19.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:19 smithi006 bash[26022]: cluster 2023-11-17T05:48:17.972925+0000 mgr.smithi006.odexej (mgr.14176) 661 : cluster [DBG] pgmap v482: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:19.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:19 smithi156 bash[29655]: cluster 2023-11-17T05:48:17.972925+0000 mgr.smithi006.odexej (mgr.14176) 661 : cluster [DBG] pgmap v482: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:20.258 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:20.258 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:20.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:20 smithi006 bash[26022]: cluster 2023-11-17T05:48:19.974314+0000 mgr.smithi006.odexej (mgr.14176) 662 : cluster [DBG] pgmap v483: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:20.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:20 smithi156 bash[29655]: cluster 2023-11-17T05:48:19.974314+0000 mgr.smithi006.odexej (mgr.14176) 662 : cluster [DBG] pgmap v483: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:20.859 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:21.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:21 smithi006 bash[26022]: audit 2023-11-17T05:48:20.246263+0000 mgr.smithi006.odexej (mgr.14176) 663 : audit [DBG] from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:21.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:21 smithi156 bash[29655]: audit 2023-11-17T05:48:20.246263+0000 mgr.smithi006.odexej (mgr.14176) 663 : audit [DBG] from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:21.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:22.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:22 smithi006 bash[26022]: cluster 2023-11-17T05:48:21.975298+0000 mgr.smithi006.odexej (mgr.14176) 664 : cluster [DBG] pgmap v484: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:22.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:22 smithi156 bash[29655]: cluster 2023-11-17T05:48:21.975298+0000 mgr.smithi006.odexej (mgr.14176) 664 : cluster [DBG] pgmap v484: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:25.115 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:25.116 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:25 smithi006 bash[26022]: cluster 2023-11-17T05:48:23.976408+0000 mgr.smithi006.odexej (mgr.14176) 665 : cluster [DBG] pgmap v485: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:25 smithi156 bash[29655]: cluster 2023-11-17T05:48:23.976408+0000 mgr.smithi006.odexej (mgr.14176) 665 : cluster [DBG] pgmap v485: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:25.750 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:26.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:26 smithi006 bash[26022]: audit 2023-11-17T05:48:25.095330+0000 mgr.smithi006.odexej (mgr.14176) 666 : audit [DBG] from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:26.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:26 smithi006 bash[26022]: cluster 2023-11-17T05:48:25.977838+0000 mgr.smithi006.odexej (mgr.14176) 667 : cluster [DBG] pgmap v486: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:26.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:26 smithi156 bash[29655]: audit 2023-11-17T05:48:25.095330+0000 mgr.smithi006.odexej (mgr.14176) 666 : audit [DBG] from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:26.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:26 smithi156 bash[29655]: cluster 2023-11-17T05:48:25.977838+0000 mgr.smithi006.odexej (mgr.14176) 667 : cluster [DBG] pgmap v486: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:26.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:29.307 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:29 smithi006 bash[26022]: cluster 2023-11-17T05:48:27.978966+0000 mgr.smithi006.odexej (mgr.14176) 668 : cluster [DBG] pgmap v487: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:29 smithi156 bash[29655]: cluster 2023-11-17T05:48:27.978966+0000 mgr.smithi006.odexej (mgr.14176) 668 : cluster [DBG] pgmap v487: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:29.771 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:30.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:30 smithi006 bash[26022]: audit 2023-11-17T05:48:29.750663+0000 mgr.smithi006.odexej (mgr.14176) 669 : audit [DBG] from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:30.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:30 smithi006 bash[26022]: cluster 2023-11-17T05:48:29.980448+0000 mgr.smithi006.odexej (mgr.14176) 670 : cluster [DBG] pgmap v488: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:30.402 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:30.479 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:30 smithi156 bash[29655]: audit 2023-11-17T05:48:29.750663+0000 mgr.smithi006.odexej (mgr.14176) 669 : audit [DBG] from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:30.479 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:30 smithi156 bash[29655]: cluster 2023-11-17T05:48:29.980448+0000 mgr.smithi006.odexej (mgr.14176) 670 : cluster [DBG] pgmap v488: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:31.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:31.948364+0000 mon.smithi006 (mon.0) 797 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: cluster 2023-11-17T05:48:31.981743+0000 mgr.smithi006.odexej (mgr.14176) 671 : cluster [DBG] pgmap v489: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:32.258384+0000 mon.smithi006 (mon.0) 798 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:32.259809+0000 mon.smithi006 (mon.0) 799 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:32.269388+0000 mon.smithi006 (mon.0) 800 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:32.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:32.279185+0000 mon.smithi006 (mon.0) 801 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:48:32.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:32 smithi006 bash[26022]: audit 2023-11-17T05:48:32.294879+0000 mon.smithi006 (mon.0) 802 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:31.948364+0000 mon.smithi006 (mon.0) 797 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: cluster 2023-11-17T05:48:31.981743+0000 mgr.smithi006.odexej (mgr.14176) 671 : cluster [DBG] pgmap v489: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:32.258384+0000 mon.smithi006 (mon.0) 798 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:32.259809+0000 mon.smithi006 (mon.0) 799 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:32.269388+0000 mon.smithi006 (mon.0) 800 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:32.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:32.279185+0000 mon.smithi006 (mon.0) 801 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:48:32.714 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:32 smithi156 bash[29655]: audit 2023-11-17T05:48:32.294879+0000 mon.smithi006 (mon.0) 802 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:33.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:33 smithi006 bash[26022]: cluster 2023-11-17T05:48:32.270409+0000 mgr.smithi006.odexej (mgr.14176) 672 : cluster [DBG] pgmap v490: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:33.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:33 smithi006 bash[26022]: cephadm 2023-11-17T05:48:32.300631+0000 mgr.smithi006.odexej (mgr.14176) 673 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ibacnl on smithi156 2023-11-17T05:48:33.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:33 smithi006 bash[26022]: cluster 2023-11-17T05:48:32.394808+0000 mon.smithi006 (mon.0) 803 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:48:33.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:33 smithi006 bash[26022]: cluster 2023-11-17T05:48:32.394883+0000 mon.smithi006 (mon.0) 804 : cluster [INF] Cluster is now healthy 2023-11-17T05:48:33.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:33 smithi156 bash[29655]: cluster 2023-11-17T05:48:32.270409+0000 mgr.smithi006.odexej (mgr.14176) 672 : cluster [DBG] pgmap v490: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:33.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:33 smithi156 bash[29655]: cephadm 2023-11-17T05:48:32.300631+0000 mgr.smithi006.odexej (mgr.14176) 673 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ibacnl on smithi156 2023-11-17T05:48:33.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:33 smithi156 bash[29655]: cluster 2023-11-17T05:48:32.394808+0000 mon.smithi006 (mon.0) 803 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:48:33.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:33 smithi156 bash[29655]: cluster 2023-11-17T05:48:32.394883+0000 mon.smithi006 (mon.0) 804 : cluster [INF] Cluster is now healthy 2023-11-17T05:48:34.650 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:34.651 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:46:24.077162Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.pomyrd on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:35.273 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:36.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:35 smithi156 bash[29655]: cluster 2023-11-17T05:48:34.271198+0000 mgr.smithi006.odexej (mgr.14176) 674 : cluster [DBG] pgmap v491: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:36.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:35 smithi156 bash[29655]: audit 2023-11-17T05:48:34.630452+0000 mgr.smithi006.odexej (mgr.14176) 675 : audit [DBG] from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:36.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:35 smithi156 bash[29655]: audit 2023-11-17T05:48:34.713327+0000 mon.smithi006 (mon.0) 805 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:36.072 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:35 smithi006 bash[26022]: cluster 2023-11-17T05:48:34.271198+0000 mgr.smithi006.odexej (mgr.14176) 674 : cluster [DBG] pgmap v491: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:36.072 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:35 smithi006 bash[26022]: audit 2023-11-17T05:48:34.630452+0000 mgr.smithi006.odexej (mgr.14176) 675 : audit [DBG] from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:36.072 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:35 smithi006 bash[26022]: audit 2023-11-17T05:48:34.713327+0000 mon.smithi006 (mon.0) 805 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:36.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:37.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:36 smithi006 bash[26022]: audit 2023-11-17T05:48:36.066843+0000 mon.smithi006 (mon.0) 806 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ibacnl"}]: dispatch 2023-11-17T05:48:37.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:36 smithi156 bash[29655]: audit 2023-11-17T05:48:36.066843+0000 mon.smithi006 (mon.0) 806 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ibacnl"}]: dispatch 2023-11-17T05:48:38.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: cephadm 2023-11-17T05:48:36.065583+0000 mgr.smithi006.odexej (mgr.14176) 676 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:48:38.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: cephadm 2023-11-17T05:48:36.066213+0000 mgr.smithi006.odexej (mgr.14176) 677 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: cephadm 2023-11-17T05:48:36.068519+0000 mgr.smithi006.odexej (mgr.14176) 678 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: cephadm 2023-11-17T05:48:36.073567+0000 mgr.smithi006.odexej (mgr.14176) 679 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.txcraq on smithi006 2023-11-17T05:48:38.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:37 smithi006 bash[26022]: cluster 2023-11-17T05:48:36.272312+0000 mgr.smithi006.odexej (mgr.14176) 680 : cluster [DBG] pgmap v492: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:38.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: cephadm 2023-11-17T05:48:36.065583+0000 mgr.smithi006.odexej (mgr.14176) 676 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:48:38.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: cephadm 2023-11-17T05:48:36.066213+0000 mgr.smithi006.odexej (mgr.14176) 677 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: cephadm 2023-11-17T05:48:36.068519+0000 mgr.smithi006.odexej (mgr.14176) 678 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ibacnl ... 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:38.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:38.190 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:38.190 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: cephadm 2023-11-17T05:48:36.073567+0000 mgr.smithi006.odexej (mgr.14176) 679 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.txcraq on smithi006 2023-11-17T05:48:38.190 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:37 smithi156 bash[29655]: cluster 2023-11-17T05:48:36.272312+0000 mgr.smithi006.odexej (mgr.14176) 680 : cluster [DBG] pgmap v492: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:39.370 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:39.370 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:28.177045Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gmbhov on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gmbhov\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gmbhov\nDeploy daemon haproxy.nfs.foo.smithi156.gmbhov ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.940688Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sfefkl on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sfefkl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sfefkl\nDeploy daemon haproxy.nfs.foo.smithi006.sfefkl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:47:31.942632Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jlggxd on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:40.010 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:39 smithi006 bash[26022]: cluster 2023-11-17T05:48:38.273120+0000 mgr.smithi006.odexej (mgr.14176) 681 : cluster [DBG] pgmap v493: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:40.073 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:39 smithi156 bash[29655]: cluster 2023-11-17T05:48:38.273120+0000 mgr.smithi006.odexej (mgr.14176) 681 : cluster [DBG] pgmap v493: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:48:41.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: audit 2023-11-17T05:48:39.350400+0000 mgr.smithi006.odexej (mgr.14176) 682 : audit [DBG] from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cephadm 2023-11-17T05:48:40.003840+0000 mgr.smithi006.odexej (mgr.14176) 683 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cephadm 2023-11-17T05:48:40.004481+0000 mgr.smithi006.odexej (mgr.14176) 684 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: audit 2023-11-17T05:48:40.005195+0000 mon.smithi006 (mon.0) 807 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.txcraq"}]: dispatch 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cephadm 2023-11-17T05:48:40.006732+0000 mgr.smithi006.odexej (mgr.14176) 685 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cephadm 2023-11-17T05:48:40.011211+0000 mgr.smithi006.odexej (mgr.14176) 686 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cephadm 2023-11-17T05:48:40.015234+0000 mgr.smithi006.odexej (mgr.14176) 687 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:41.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: cluster 2023-11-17T05:48:40.016800+0000 mgr.smithi006.odexej (mgr.14176) 688 : cluster [DBG] pgmap v494: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:48:41.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:40 smithi006 bash[26022]: audit 2023-11-17T05:48:40.397059+0000 mon.smithi006 (mon.0) 808 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: audit 2023-11-17T05:48:39.350400+0000 mgr.smithi006.odexej (mgr.14176) 682 : audit [DBG] from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cephadm 2023-11-17T05:48:40.003840+0000 mgr.smithi006.odexej (mgr.14176) 683 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cephadm 2023-11-17T05:48:40.004481+0000 mgr.smithi006.odexej (mgr.14176) 684 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: audit 2023-11-17T05:48:40.005195+0000 mon.smithi006 (mon.0) 807 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.txcraq"}]: dispatch 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cephadm 2023-11-17T05:48:40.006732+0000 mgr.smithi006.odexej (mgr.14176) 685 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.txcraq ... 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cephadm 2023-11-17T05:48:40.011211+0000 mgr.smithi006.odexej (mgr.14176) 686 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cephadm 2023-11-17T05:48:40.015234+0000 mgr.smithi006.odexej (mgr.14176) 687 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:48:41.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: cluster 2023-11-17T05:48:40.016800+0000 mgr.smithi006.odexej (mgr.14176) 688 : cluster [DBG] pgmap v494: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:48:41.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:40 smithi156 bash[29655]: audit 2023-11-17T05:48:40.397059+0000 mon.smithi006 (mon.0) 808 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:42.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:41 smithi006 bash[26022]: cluster 2023-11-17T05:48:40.731844+0000 mon.smithi006 (mon.0) 809 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:48:42.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:41 smithi156 bash[29655]: cluster 2023-11-17T05:48:40.731844+0000 mon.smithi006 (mon.0) 809 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:48:42.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:42 smithi006 bash[26022]: cluster 2023-11-17T05:48:42.017773+0000 mgr.smithi006.odexej (mgr.14176) 689 : cluster [DBG] pgmap v495: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:48:43.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:42 smithi156 bash[29655]: cluster 2023-11-17T05:48:42.017773+0000 mgr.smithi006.odexej (mgr.14176) 689 : cluster [DBG] pgmap v495: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:48:44.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:44 smithi006 bash[26022]: cluster 2023-11-17T05:48:44.018716+0000 mgr.smithi006.odexej (mgr.14176) 690 : cluster [DBG] pgmap v496: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T05:48:44.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:44.337 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:44.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:44 smithi156 bash[29655]: cluster 2023-11-17T05:48:44.018716+0000 mgr.smithi006.odexej (mgr.14176) 690 : cluster [DBG] pgmap v496: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T05:48:44.981 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:45.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:45 smithi006 bash[26022]: audit 2023-11-17T05:48:44.317392+0000 mgr.smithi006.odexej (mgr.14176) 691 : audit [DBG] from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:45.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:45 smithi156 bash[29655]: audit 2023-11-17T05:48:44.317392+0000 mgr.smithi006.odexej (mgr.14176) 691 : audit [DBG] from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:45.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:46.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:46 smithi006 bash[26022]: cluster 2023-11-17T05:48:46.020194+0000 mgr.smithi006.odexej (mgr.14176) 692 : cluster [DBG] pgmap v497: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:46.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:46 smithi156 bash[29655]: cluster 2023-11-17T05:48:46.020194+0000 mgr.smithi006.odexej (mgr.14176) 692 : cluster [DBG] pgmap v497: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:48.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:48 smithi006 bash[26022]: cluster 2023-11-17T05:48:48.021293+0000 mgr.smithi006.odexej (mgr.14176) 693 : cluster [DBG] pgmap v498: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:48 smithi156 bash[29655]: cluster 2023-11-17T05:48:48.021293+0000 mgr.smithi006.odexej (mgr.14176) 693 : cluster [DBG] pgmap v498: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:48.977 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:49.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:49 smithi006 bash[26022]: audit 2023-11-17T05:48:48.960607+0000 mgr.smithi006.odexej (mgr.14176) 694 : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:49.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:49 smithi156 bash[29655]: audit 2023-11-17T05:48:48.960607+0000 mgr.smithi006.odexej (mgr.14176) 694 : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:49.559 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:50.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:50.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:50 smithi006 bash[26022]: audit 2023-11-17T05:48:49.715022+0000 mon.smithi006 (mon.0) 810 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:50.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:50 smithi006 bash[26022]: cluster 2023-11-17T05:48:50.022809+0000 mgr.smithi006.odexej (mgr.14176) 695 : cluster [DBG] pgmap v499: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:51.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:50 smithi156 bash[29655]: audit 2023-11-17T05:48:49.715022+0000 mon.smithi006 (mon.0) 810 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:48:51.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:50 smithi156 bash[29655]: cluster 2023-11-17T05:48:50.022809+0000 mgr.smithi006.odexej (mgr.14176) 695 : cluster [DBG] pgmap v499: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:48:52.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:52 smithi006 bash[26022]: cluster 2023-11-17T05:48:52.023991+0000 mgr.smithi006.odexej (mgr.14176) 696 : cluster [DBG] pgmap v500: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:52.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:52 smithi156 bash[29655]: cluster 2023-11-17T05:48:52.023991+0000 mgr.smithi006.odexej (mgr.14176) 696 : cluster [DBG] pgmap v500: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:53.625 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:53.625 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:54.222 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:54.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:54 smithi006 bash[26022]: audit 2023-11-17T05:48:53.605059+0000 mgr.smithi006.odexej (mgr.14176) 697 : audit [DBG] from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:54.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:54 smithi006 bash[26022]: cluster 2023-11-17T05:48:54.025185+0000 mgr.smithi006.odexej (mgr.14176) 698 : cluster [DBG] pgmap v501: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:54.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:54 smithi156 bash[29655]: audit 2023-11-17T05:48:53.605059+0000 mgr.smithi006.odexej (mgr.14176) 697 : audit [DBG] from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:54.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:54 smithi156 bash[29655]: cluster 2023-11-17T05:48:54.025185+0000 mgr.smithi006.odexej (mgr.14176) 698 : cluster [DBG] pgmap v501: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:48:55.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:48:56.398 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:56 smithi006 bash[26022]: cluster 2023-11-17T05:48:56.026659+0000 mgr.smithi006.odexej (mgr.14176) 699 : cluster [DBG] pgmap v502: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:56 smithi156 bash[29655]: cluster 2023-11-17T05:48:56.026659+0000 mgr.smithi006.odexej (mgr.14176) 699 : cluster [DBG] pgmap v502: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:58.232 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:48:58.232 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:48:58.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:58 smithi006 bash[26022]: cluster 2023-11-17T05:48:58.027327+0000 mgr.smithi006.odexej (mgr.14176) 700 : cluster [DBG] pgmap v503: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:58.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:58 smithi156 bash[29655]: cluster 2023-11-17T05:48:58.027327+0000 mgr.smithi006.odexej (mgr.14176) 700 : cluster [DBG] pgmap v503: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:48:58.867 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:48:59.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:48:59 smithi006 bash[26022]: audit 2023-11-17T05:48:58.220184+0000 mgr.smithi006.odexej (mgr.14176) 701 : audit [DBG] from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:59.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:48:59 smithi156 bash[29655]: audit 2023-11-17T05:48:58.220184+0000 mgr.smithi006.odexej (mgr.14176) 701 : audit [DBG] from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:48:59.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:00.193 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:00 smithi006 bash[26022]: cluster 2023-11-17T05:49:00.028939+0000 mgr.smithi006.odexej (mgr.14176) 702 : cluster [DBG] pgmap v504: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:00.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:00 smithi156 bash[29655]: cluster 2023-11-17T05:49:00.028939+0000 mgr.smithi006.odexej (mgr.14176) 702 : cluster [DBG] pgmap v504: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:02.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:02 smithi156 bash[29655]: cluster 2023-11-17T05:49:02.030157+0000 mgr.smithi006.odexej (mgr.14176) 703 : cluster [DBG] pgmap v505: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.963 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:02 smithi006 bash[26022]: cluster 2023-11-17T05:49:02.030157+0000 mgr.smithi006.odexej (mgr.14176) 703 : cluster [DBG] pgmap v505: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.964 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:14.964 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:03 smithi156 bash[29655]: audit 2023-11-17T05:49:02.916457+0000 mgr.smithi006.odexej (mgr.14176) 704 : audit [DBG] from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:04 smithi156 bash[29655]: cluster 2023-11-17T05:49:04.031503+0000 mgr.smithi006.odexej (mgr.14176) 705 : cluster [DBG] pgmap v506: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:06 smithi156 bash[29655]: cluster 2023-11-17T05:49:06.032985+0000 mgr.smithi006.odexej (mgr.14176) 706 : cluster [DBG] pgmap v507: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:08 smithi156 bash[29655]: cluster 2023-11-17T05:49:08.033996+0000 mgr.smithi006.odexej (mgr.14176) 707 : cluster [DBG] pgmap v508: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:10 smithi156 bash[29655]: cluster 2023-11-17T05:49:10.035414+0000 mgr.smithi006.odexej (mgr.14176) 708 : cluster [DBG] pgmap v509: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:12 smithi156 bash[29655]: cluster 2023-11-17T05:49:12.036475+0000 mgr.smithi006.odexej (mgr.14176) 709 : cluster [DBG] pgmap v510: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:14 smithi156 bash[29655]: cluster 2023-11-17T05:49:14.037653+0000 mgr.smithi006.odexej (mgr.14176) 710 : cluster [DBG] pgmap v511: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:14.965 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:14.968 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:03 smithi006 bash[26022]: audit 2023-11-17T05:49:02.916457+0000 mgr.smithi006.odexej (mgr.14176) 704 : audit [DBG] from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:15.124 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:04 smithi006 bash[26022]: cluster 2023-11-17T05:49:04.031503+0000 mgr.smithi006.odexej (mgr.14176) 705 : cluster [DBG] pgmap v506: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:15.124 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:06 smithi006 bash[26022]: cluster 2023-11-17T05:49:06.032985+0000 mgr.smithi006.odexej (mgr.14176) 706 : cluster [DBG] pgmap v507: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:15.124 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:08 smithi006 bash[26022]: cluster 2023-11-17T05:49:08.033996+0000 mgr.smithi006.odexej (mgr.14176) 707 : cluster [DBG] pgmap v508: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:15.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:10 smithi006 bash[26022]: cluster 2023-11-17T05:49:10.035414+0000 mgr.smithi006.odexej (mgr.14176) 708 : cluster [DBG] pgmap v509: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:15.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:12 smithi006 bash[26022]: cluster 2023-11-17T05:49:12.036475+0000 mgr.smithi006.odexej (mgr.14176) 709 : cluster [DBG] pgmap v510: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:15.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:14 smithi006 bash[26022]: cluster 2023-11-17T05:49:14.037653+0000 mgr.smithi006.odexej (mgr.14176) 710 : cluster [DBG] pgmap v511: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:15.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:16.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:16 smithi006 bash[26022]: cluster 2023-11-17T05:49:16.038916+0000 mgr.smithi006.odexej (mgr.14176) 711 : cluster [DBG] pgmap v512: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:16.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:16 smithi156 bash[29655]: cluster 2023-11-17T05:49:16.038916+0000 mgr.smithi006.odexej (mgr.14176) 711 : cluster [DBG] pgmap v512: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:18.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:18 smithi006 bash[26022]: cluster 2023-11-17T05:49:18.040014+0000 mgr.smithi006.odexej (mgr.14176) 712 : cluster [DBG] pgmap v513: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:18.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:18 smithi156 bash[29655]: cluster 2023-11-17T05:49:18.040014+0000 mgr.smithi006.odexej (mgr.14176) 712 : cluster [DBG] pgmap v513: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:19.221 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:19.221 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:19.835 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:20.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:20 smithi006 bash[26022]: audit 2023-11-17T05:49:19.206514+0000 mgr.smithi006.odexej (mgr.14176) 713 : audit [DBG] from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:20.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:20 smithi006 bash[26022]: cluster 2023-11-17T05:49:20.041465+0000 mgr.smithi006.odexej (mgr.14176) 714 : cluster [DBG] pgmap v514: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:20.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:20 smithi156 bash[29655]: audit 2023-11-17T05:49:19.206514+0000 mgr.smithi006.odexej (mgr.14176) 713 : audit [DBG] from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:20.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:20 smithi156 bash[29655]: cluster 2023-11-17T05:49:20.041465+0000 mgr.smithi006.odexej (mgr.14176) 714 : cluster [DBG] pgmap v514: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:20.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:22.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:22 smithi156 bash[29655]: cluster 2023-11-17T05:49:22.042493+0000 mgr.smithi006.odexej (mgr.14176) 715 : cluster [DBG] pgmap v515: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:22.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:22 smithi006 bash[26022]: cluster 2023-11-17T05:49:22.042493+0000 mgr.smithi006.odexej (mgr.14176) 715 : cluster [DBG] pgmap v515: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:23.940 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:23.940 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:24.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:24 smithi006 bash[26022]: audit 2023-11-17T05:49:23.922686+0000 mgr.smithi006.odexej (mgr.14176) 716 : audit [DBG] from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:24.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:24 smithi006 bash[26022]: cluster 2023-11-17T05:49:24.043802+0000 mgr.smithi006.odexej (mgr.14176) 717 : cluster [DBG] pgmap v516: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:24.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:24 smithi156 bash[29655]: audit 2023-11-17T05:49:23.922686+0000 mgr.smithi006.odexej (mgr.14176) 716 : audit [DBG] from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:24.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:24 smithi156 bash[29655]: cluster 2023-11-17T05:49:24.043802+0000 mgr.smithi006.odexej (mgr.14176) 717 : cluster [DBG] pgmap v516: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:24.548 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:25.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:27.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:27 smithi156 bash[29655]: cluster 2023-11-17T05:49:26.045274+0000 mgr.smithi006.odexej (mgr.14176) 718 : cluster [DBG] pgmap v517: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:27.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:27 smithi006 bash[26022]: cluster 2023-11-17T05:49:26.045274+0000 mgr.smithi006.odexej (mgr.14176) 718 : cluster [DBG] pgmap v517: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:28.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:28.661 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:29.246 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:29.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:29 smithi006 bash[26022]: cluster 2023-11-17T05:49:28.046286+0000 mgr.smithi006.odexej (mgr.14176) 719 : cluster [DBG] pgmap v518: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:29.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:29 smithi006 bash[26022]: audit 2023-11-17T05:49:28.643437+0000 mgr.smithi006.odexej (mgr.14176) 720 : audit [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:29.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:29 smithi156 bash[29655]: cluster 2023-11-17T05:49:28.046286+0000 mgr.smithi006.odexej (mgr.14176) 719 : cluster [DBG] pgmap v518: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:29.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:29 smithi156 bash[29655]: audit 2023-11-17T05:49:28.643437+0000 mgr.smithi006.odexej (mgr.14176) 720 : audit [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:30.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:31.640 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:31 smithi006 bash[26022]: cluster 2023-11-17T05:49:30.047739+0000 mgr.smithi006.odexej (mgr.14176) 721 : cluster [DBG] pgmap v519: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:31.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:31 smithi156 bash[29655]: cluster 2023-11-17T05:49:30.047739+0000 mgr.smithi006.odexej (mgr.14176) 721 : cluster [DBG] pgmap v519: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:33.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:33.307 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:33.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:33 smithi006 bash[26022]: cluster 2023-11-17T05:49:32.048793+0000 mgr.smithi006.odexej (mgr.14176) 722 : cluster [DBG] pgmap v520: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:33.895 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:33.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:33 smithi156 bash[29655]: cluster 2023-11-17T05:49:32.048793+0000 mgr.smithi006.odexej (mgr.14176) 722 : cluster [DBG] pgmap v520: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:34.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:34.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:34 smithi006 bash[26022]: audit 2023-11-17T05:49:33.289812+0000 mgr.smithi006.odexej (mgr.14176) 723 : audit [DBG] from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:34.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:34 smithi156 bash[29655]: audit 2023-11-17T05:49:33.289812+0000 mgr.smithi006.odexej (mgr.14176) 723 : audit [DBG] from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:35.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:35 smithi006 bash[26022]: cluster 2023-11-17T05:49:34.049981+0000 mgr.smithi006.odexej (mgr.14176) 724 : cluster [DBG] pgmap v521: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:35.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:35 smithi156 bash[29655]: cluster 2023-11-17T05:49:34.049981+0000 mgr.smithi006.odexej (mgr.14176) 724 : cluster [DBG] pgmap v521: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:49:37.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:37 smithi006 bash[26022]: cluster 2023-11-17T05:49:36.051793+0000 mgr.smithi006.odexej (mgr.14176) 725 : cluster [DBG] pgmap v522: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:37.928 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:37.928 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:37.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:37 smithi156 bash[29655]: cluster 2023-11-17T05:49:36.051793+0000 mgr.smithi006.odexej (mgr.14176) 725 : cluster [DBG] pgmap v522: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:38.491 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:38.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:38 smithi006 bash[26022]: audit 2023-11-17T05:49:37.912492+0000 mgr.smithi006.odexej (mgr.14176) 726 : audit [DBG] from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:38.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:38 smithi156 bash[29655]: audit 2023-11-17T05:49:37.912492+0000 mgr.smithi006.odexej (mgr.14176) 726 : audit [DBG] from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:39.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:39.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:39 smithi006 bash[26022]: cluster 2023-11-17T05:49:38.052849+0000 mgr.smithi006.odexej (mgr.14176) 727 : cluster [DBG] pgmap v523: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:39.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:39 smithi156 bash[29655]: cluster 2023-11-17T05:49:38.052849+0000 mgr.smithi006.odexej (mgr.14176) 727 : cluster [DBG] pgmap v523: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.019779+0000 mon.smithi006 (mon.0) 811 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.334924+0000 mon.smithi006 (mon.0) 812 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.336010+0000 mon.smithi006 (mon.0) 813 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.343368+0000 mon.smithi006 (mon.0) 814 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.352319+0000 mon.smithi006 (mon.0) 815 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:49:40.840 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:40 smithi006 bash[26022]: audit 2023-11-17T05:49:40.366801+0000 mon.smithi006 (mon.0) 816 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.019779+0000 mon.smithi006 (mon.0) 811 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.334924+0000 mon.smithi006 (mon.0) 812 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.336010+0000 mon.smithi006 (mon.0) 813 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.343368+0000 mon.smithi006 (mon.0) 814 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.352319+0000 mon.smithi006 (mon.0) 815 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:49:40.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:40 smithi156 bash[29655]: audit 2023-11-17T05:49:40.366801+0000 mon.smithi006 (mon.0) 816 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:41.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:41 smithi006 bash[26022]: cluster 2023-11-17T05:49:40.054137+0000 mgr.smithi006.odexej (mgr.14176) 728 : cluster [DBG] pgmap v524: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:41.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:41 smithi006 bash[26022]: cluster 2023-11-17T05:49:40.344172+0000 mgr.smithi006.odexej (mgr.14176) 729 : cluster [DBG] pgmap v525: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:41.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:41 smithi006 bash[26022]: cephadm 2023-11-17T05:49:40.370862+0000 mgr.smithi006.odexej (mgr.14176) 730 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qjyrxf on smithi156 2023-11-17T05:49:41.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:41 smithi006 bash[26022]: cluster 2023-11-17T05:49:40.464345+0000 mon.smithi006 (mon.0) 817 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:49:41.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:41 smithi006 bash[26022]: cluster 2023-11-17T05:49:40.464437+0000 mon.smithi006 (mon.0) 818 : cluster [INF] Cluster is now healthy 2023-11-17T05:49:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:41 smithi156 bash[29655]: cluster 2023-11-17T05:49:40.054137+0000 mgr.smithi006.odexej (mgr.14176) 728 : cluster [DBG] pgmap v524: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:49:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:41 smithi156 bash[29655]: cluster 2023-11-17T05:49:40.344172+0000 mgr.smithi006.odexej (mgr.14176) 729 : cluster [DBG] pgmap v525: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:41 smithi156 bash[29655]: cephadm 2023-11-17T05:49:40.370862+0000 mgr.smithi006.odexej (mgr.14176) 730 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qjyrxf on smithi156 2023-11-17T05:49:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:41 smithi156 bash[29655]: cluster 2023-11-17T05:49:40.464345+0000 mon.smithi006 (mon.0) 817 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:49:41.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:41 smithi156 bash[29655]: cluster 2023-11-17T05:49:40.464437+0000 mon.smithi006 (mon.0) 818 : cluster [INF] Cluster is now healthy 2023-11-17T05:49:42.624 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:42.624 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:47:31.944314Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iitqqy on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:43.208 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:43.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:43 smithi156 bash[29655]: cluster 2023-11-17T05:49:42.344693+0000 mgr.smithi006.odexej (mgr.14176) 731 : cluster [DBG] pgmap v526: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:43.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:43 smithi156 bash[29655]: audit 2023-11-17T05:49:42.606734+0000 mgr.smithi006.odexej (mgr.14176) 732 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:43.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:43 smithi006 bash[26022]: cluster 2023-11-17T05:49:42.344693+0000 mgr.smithi006.odexej (mgr.14176) 731 : cluster [DBG] pgmap v526: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:43.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:43 smithi006 bash[26022]: audit 2023-11-17T05:49:42.606734+0000 mgr.smithi006.odexej (mgr.14176) 732 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:44.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:44 smithi006 bash[26022]: audit 2023-11-17T05:49:44.096495+0000 mon.smithi006 (mon.0) 819 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qjyrxf"}]: dispatch 2023-11-17T05:49:44.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:44 smithi156 bash[29655]: audit 2023-11-17T05:49:44.096495+0000 mon.smithi006 (mon.0) 819 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qjyrxf"}]: dispatch 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: cephadm 2023-11-17T05:49:44.095209+0000 mgr.smithi006.odexej (mgr.14176) 733 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.692 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: cephadm 2023-11-17T05:49:44.095761+0000 mgr.smithi006.odexej (mgr.14176) 734 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: cephadm 2023-11-17T05:49:44.098240+0000 mgr.smithi006.odexej (mgr.14176) 735 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.693 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.694 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: cephadm 2023-11-17T05:49:44.103437+0000 mgr.smithi006.odexej (mgr.14176) 736 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.zweera on smithi006 2023-11-17T05:49:45.697 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:45 smithi006 bash[26022]: cluster 2023-11-17T05:49:44.345698+0000 mgr.smithi006.odexej (mgr.14176) 737 : cluster [DBG] pgmap v527: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:45.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: cephadm 2023-11-17T05:49:44.095209+0000 mgr.smithi006.odexej (mgr.14176) 733 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: cephadm 2023-11-17T05:49:44.095761+0000 mgr.smithi006.odexej (mgr.14176) 734 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: cephadm 2023-11-17T05:49:44.098240+0000 mgr.smithi006.odexej (mgr.14176) 735 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:49:45.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: cephadm 2023-11-17T05:49:44.103437+0000 mgr.smithi006.odexej (mgr.14176) 736 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.zweera on smithi006 2023-11-17T05:49:45.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:45 smithi156 bash[29655]: cluster 2023-11-17T05:49:44.345698+0000 mgr.smithi006.odexej (mgr.14176) 737 : cluster [DBG] pgmap v527: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:47.391 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:47.391 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:48:36.068282Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ibacnl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ibacnl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ibacnl\nDeploy daemon haproxy.nfs.foo.smithi156.ibacnl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.006451Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.txcraq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-txcraq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.txcraq\nDeploy daemon haproxy.nfs.foo.smithi006.txcraq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:48:40.010988Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vjpman on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:44.098008Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\nDeploy daemon haproxy.nfs.foo.smithi156.qjyrxf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:47.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:47 smithi006 bash[26022]: cluster 2023-11-17T05:49:46.346703+0000 mgr.smithi006.odexej (mgr.14176) 738 : cluster [DBG] pgmap v528: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:47.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:47 smithi156 bash[29655]: cluster 2023-11-17T05:49:46.346703+0000 mgr.smithi006.odexej (mgr.14176) 738 : cluster [DBG] pgmap v528: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:49:48.132 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:48.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:48 smithi006 bash[26022]: audit 2023-11-17T05:49:47.373478+0000 mgr.smithi006.odexej (mgr.14176) 739 : audit [DBG] from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:48.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:48 smithi006 bash[26022]: audit 2023-11-17T05:49:48.066487+0000 mon.smithi006 (mon.0) 820 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.zweera"}]: dispatch 2023-11-17T05:49:48.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:48 smithi156 bash[29655]: audit 2023-11-17T05:49:47.373478+0000 mgr.smithi006.odexej (mgr.14176) 739 : audit [DBG] from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:48.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:48 smithi156 bash[29655]: audit 2023-11-17T05:49:48.066487+0000 mon.smithi006 (mon.0) 820 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.zweera"}]: dispatch 2023-11-17T05:49:49.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cephadm 2023-11-17T05:49:48.065142+0000 mgr.smithi006.odexej (mgr.14176) 740 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cephadm 2023-11-17T05:49:48.065799+0000 mgr.smithi006.odexej (mgr.14176) 741 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cephadm 2023-11-17T05:49:48.067923+0000 mgr.smithi006.odexej (mgr.14176) 742 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cephadm 2023-11-17T05:49:48.069872+0000 mgr.smithi006.odexej (mgr.14176) 743 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cephadm 2023-11-17T05:49:48.071488+0000 mgr.smithi006.odexej (mgr.14176) 744 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cluster 2023-11-17T05:49:48.072671+0000 mgr.smithi006.odexej (mgr.14176) 745 : cluster [DBG] pgmap v529: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:49:49.923 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:49 smithi006 bash[26022]: cluster 2023-11-17T05:49:48.504529+0000 mon.smithi006 (mon.0) 821 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:49:49.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cephadm 2023-11-17T05:49:48.065142+0000 mgr.smithi006.odexej (mgr.14176) 740 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:49:49.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cephadm 2023-11-17T05:49:48.065799+0000 mgr.smithi006.odexej (mgr.14176) 741 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cephadm 2023-11-17T05:49:48.067923+0000 mgr.smithi006.odexej (mgr.14176) 742 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:49:49.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:49:49.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:49:49.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cephadm 2023-11-17T05:49:48.069872+0000 mgr.smithi006.odexej (mgr.14176) 743 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:49:49.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cephadm 2023-11-17T05:49:48.071488+0000 mgr.smithi006.odexej (mgr.14176) 744 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:49:49.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cluster 2023-11-17T05:49:48.072671+0000 mgr.smithi006.odexej (mgr.14176) 745 : cluster [DBG] pgmap v529: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:49:49.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:49 smithi156 bash[29655]: cluster 2023-11-17T05:49:48.504529+0000 mon.smithi006 (mon.0) 821 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:49:51.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:51 smithi006 bash[26022]: cluster 2023-11-17T05:49:50.074015+0000 mgr.smithi006.odexej (mgr.14176) 746 : cluster [DBG] pgmap v530: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:49:51.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:51 smithi006 bash[26022]: audit 2023-11-17T05:49:50.455041+0000 mon.smithi006 (mon.0) 822 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:51.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:51 smithi156 bash[29655]: cluster 2023-11-17T05:49:50.074015+0000 mgr.smithi006.odexej (mgr.14176) 746 : cluster [DBG] pgmap v530: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T05:49:51.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:51 smithi156 bash[29655]: audit 2023-11-17T05:49:50.455041+0000 mon.smithi006 (mon.0) 822 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:49:52.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:52.297 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:44.098008Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\nDeploy daemon haproxy.nfs.foo.smithi156.qjyrxf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.067791Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\nDeploy daemon haproxy.nfs.foo.smithi006.zweera ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.069759Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:52.975 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:53.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:53 smithi006 bash[26022]: cluster 2023-11-17T05:49:52.074794+0000 mgr.smithi006.odexej (mgr.14176) 747 : cluster [DBG] pgmap v531: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:49:53.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:53 smithi006 bash[26022]: audit 2023-11-17T05:49:52.285163+0000 mgr.smithi006.odexej (mgr.14176) 748 : audit [DBG] from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:53.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:53 smithi156 bash[29655]: cluster 2023-11-17T05:49:52.074794+0000 mgr.smithi006.odexej (mgr.14176) 747 : cluster [DBG] pgmap v531: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:49:53.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:53 smithi156 bash[29655]: audit 2023-11-17T05:49:52.285163+0000 mgr.smithi006.odexej (mgr.14176) 748 : audit [DBG] from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:53.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:55.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:55 smithi006 bash[26022]: cluster 2023-11-17T05:49:54.075964+0000 mgr.smithi006.odexej (mgr.14176) 749 : cluster [DBG] pgmap v532: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T05:49:55.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:55 smithi156 bash[29655]: cluster 2023-11-17T05:49:54.075964+0000 mgr.smithi006.odexej (mgr.14176) 749 : cluster [DBG] pgmap v532: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T05:49:57.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:49:57.080 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:44.098008Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\nDeploy daemon haproxy.nfs.foo.smithi156.qjyrxf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.067791Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\nDeploy daemon haproxy.nfs.foo.smithi006.zweera ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.069759Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:49:57.652 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:49:57.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:57 smithi006 bash[26022]: cluster 2023-11-17T05:49:56.077370+0000 mgr.smithi006.odexej (mgr.14176) 750 : cluster [DBG] pgmap v533: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:49:57.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:57 smithi156 bash[29655]: cluster 2023-11-17T05:49:56.077370+0000 mgr.smithi006.odexej (mgr.14176) 750 : cluster [DBG] pgmap v533: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:49:58.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:49:58.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:58 smithi006 bash[26022]: audit 2023-11-17T05:49:57.061687+0000 mgr.smithi006.odexej (mgr.14176) 751 : audit [DBG] from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:58.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:58 smithi156 bash[29655]: audit 2023-11-17T05:49:57.061687+0000 mgr.smithi006.odexej (mgr.14176) 751 : audit [DBG] from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:49:59.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:49:59 smithi006 bash[26022]: cluster 2023-11-17T05:49:58.078455+0000 mgr.smithi006.odexej (mgr.14176) 752 : cluster [DBG] pgmap v534: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:49:59.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:49:59 smithi156 bash[29655]: cluster 2023-11-17T05:49:58.078455+0000 mgr.smithi006.odexej (mgr.14176) 752 : cluster [DBG] pgmap v534: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000164+0000 mon.smithi006 (mon.0) 823 : cluster [WRN] Health detail: HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000247+0000 mon.smithi006 (mon.0) 824 : cluster [WRN] [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 4 daemon(s) 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000297+0000 mon.smithi006 (mon.0) 825 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000341+0000 mon.smithi006 (mon.0) 826 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000380+0000 mon.smithi006 (mon.0) 827 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000411+0000 mon.smithi006 (mon.0) 828 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000448+0000 mon.smithi006 (mon.0) 829 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000472+0000 mon.smithi006 (mon.0) 830 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000495+0000 mon.smithi006 (mon.0) 831 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000523+0000 mon.smithi006 (mon.0) 832 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000552+0000 mon.smithi006 (mon.0) 833 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000576+0000 mon.smithi006 (mon.0) 834 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000605+0000 mon.smithi006 (mon.0) 835 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000634+0000 mon.smithi006 (mon.0) 836 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000663+0000 mon.smithi006 (mon.0) 837 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000693+0000 mon.smithi006 (mon.0) 838 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000722+0000 mon.smithi006 (mon.0) 839 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000746+0000 mon.smithi006 (mon.0) 840 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000773+0000 mon.smithi006 (mon.0) 841 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000802+0000 mon.smithi006 (mon.0) 842 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000833+0000 mon.smithi006 (mon.0) 843 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000861+0000 mon.smithi006 (mon.0) 844 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000892+0000 mon.smithi006 (mon.0) 845 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000922+0000 mon.smithi006 (mon.0) 846 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000949+0000 mon.smithi006 (mon.0) 847 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000972+0000 mon.smithi006 (mon.0) 848 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.000996+0000 mon.smithi006 (mon.0) 849 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:50:00.922 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:00 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.001021+0000 mon.smithi006 (mon.0) 850 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000164+0000 mon.smithi006 (mon.0) 823 : cluster [WRN] Health detail: HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000247+0000 mon.smithi006 (mon.0) 824 : cluster [WRN] [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 4 daemon(s) 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000297+0000 mon.smithi006 (mon.0) 825 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000341+0000 mon.smithi006 (mon.0) 826 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000380+0000 mon.smithi006 (mon.0) 827 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000411+0000 mon.smithi006 (mon.0) 828 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000448+0000 mon.smithi006 (mon.0) 829 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000472+0000 mon.smithi006 (mon.0) 830 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000495+0000 mon.smithi006 (mon.0) 831 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi156.qjyrxf ... 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000523+0000 mon.smithi006 (mon.0) 832 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000552+0000 mon.smithi006 (mon.0) 833 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000576+0000 mon.smithi006 (mon.0) 834 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000605+0000 mon.smithi006 (mon.0) 835 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000634+0000 mon.smithi006 (mon.0) 836 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000663+0000 mon.smithi006 (mon.0) 837 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000693+0000 mon.smithi006 (mon.0) 838 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000722+0000 mon.smithi006 (mon.0) 839 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000746+0000 mon.smithi006 (mon.0) 840 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000773+0000 mon.smithi006 (mon.0) 841 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000802+0000 mon.smithi006 (mon.0) 842 : cluster [WRN] /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000833+0000 mon.smithi006 (mon.0) 843 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi006.zweera ... 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000861+0000 mon.smithi006 (mon.0) 844 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000892+0000 mon.smithi006 (mon.0) 845 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000922+0000 mon.smithi006 (mon.0) 846 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000949+0000 mon.smithi006 (mon.0) 847 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000972+0000 mon.smithi006 (mon.0) 848 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.000996+0000 mon.smithi006 (mon.0) 849 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:50:00.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:00 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.001021+0000 mon.smithi006 (mon.0) 850 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:50:01.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:50:01.892 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:48:40.015007Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.kewmxm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:44.098008Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qjyrxf on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qjyrxf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qjyrxf\nDeploy daemon haproxy.nfs.foo.smithi156.qjyrxf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.067791Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.zweera on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zweera\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zweera\nDeploy daemon haproxy.nfs.foo.smithi006.zweera ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:49:48.069759Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.xjbilv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:50:01.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:01 smithi006 bash[26022]: cluster 2023-11-17T05:50:00.079921+0000 mgr.smithi006.odexej (mgr.14176) 753 : cluster [DBG] pgmap v535: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:50:01.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:01 smithi156 bash[29655]: cluster 2023-11-17T05:50:00.079921+0000 mgr.smithi006.odexej (mgr.14176) 753 : cluster [DBG] pgmap v535: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:50:02.517 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:50:02.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:02 smithi006 bash[26022]: audit 2023-11-17T05:50:01.879093+0000 mgr.smithi006.odexej (mgr.14176) 754 : audit [DBG] from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:50:02.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:02 smithi156 bash[29655]: audit 2023-11-17T05:50:01.879093+0000 mgr.smithi006.odexej (mgr.14176) 754 : audit [DBG] from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:50:03.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:19.344 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:03 smithi156 bash[29655]: cluster 2023-11-17T05:50:02.080924+0000 mgr.smithi006.odexej (mgr.14176) 755 : cluster [DBG] pgmap v536: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.344 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:05 smithi156 bash[29655]: cluster 2023-11-17T05:50:04.082116+0000 mgr.smithi006.odexej (mgr.14176) 756 : cluster [DBG] pgmap v537: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.344 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:07 smithi156 bash[29655]: cluster 2023-11-17T05:50:06.083498+0000 mgr.smithi006.odexej (mgr.14176) 757 : cluster [DBG] pgmap v538: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.344 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:09 smithi156 bash[29655]: cluster 2023-11-17T05:50:08.084525+0000 mgr.smithi006.odexej (mgr.14176) 758 : cluster [DBG] pgmap v539: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.344 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:11 smithi156 bash[29655]: cluster 2023-11-17T05:50:10.085955+0000 mgr.smithi006.odexej (mgr.14176) 759 : cluster [DBG] pgmap v540: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:13 smithi156 bash[29655]: cluster 2023-11-17T05:50:12.087008+0000 mgr.smithi006.odexej (mgr.14176) 760 : cluster [DBG] pgmap v541: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:15 smithi156 bash[29655]: cluster 2023-11-17T05:50:14.088311+0000 mgr.smithi006.odexej (mgr.14176) 761 : cluster [DBG] pgmap v542: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:17 smithi156 bash[29655]: cluster 2023-11-17T05:50:16.089687+0000 mgr.smithi006.odexej (mgr.14176) 762 : cluster [DBG] pgmap v543: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:19 smithi156 bash[29655]: cluster 2023-11-17T05:50:18.090674+0000 mgr.smithi006.odexej (mgr.14176) 763 : cluster [DBG] pgmap v544: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:21 smithi156 bash[29655]: cluster 2023-11-17T05:50:20.092063+0000 mgr.smithi006.odexej (mgr.14176) 764 : cluster [DBG] pgmap v545: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.345 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:23 smithi156 bash[29655]: cluster 2023-11-17T05:50:22.093060+0000 mgr.smithi006.odexej (mgr.14176) 765 : cluster [DBG] pgmap v546: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.346 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:25 smithi156 bash[29655]: cluster 2023-11-17T05:50:24.094244+0000 mgr.smithi006.odexej (mgr.14176) 766 : cluster [DBG] pgmap v547: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.346 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:27 smithi156 bash[29655]: cluster 2023-11-17T05:50:26.095744+0000 mgr.smithi006.odexej (mgr.14176) 767 : cluster [DBG] pgmap v548: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.346 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:29 smithi156 bash[29655]: cluster 2023-11-17T05:50:28.096903+0000 mgr.smithi006.odexej (mgr.14176) 768 : cluster [DBG] pgmap v549: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:31 smithi156 bash[29655]: cluster 2023-11-17T05:50:30.098339+0000 mgr.smithi006.odexej (mgr.14176) 769 : cluster [DBG] pgmap v550: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:33 smithi156 bash[29655]: cluster 2023-11-17T05:50:32.099377+0000 mgr.smithi006.odexej (mgr.14176) 770 : cluster [DBG] pgmap v551: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:35 smithi156 bash[29655]: cluster 2023-11-17T05:50:34.100516+0000 mgr.smithi006.odexej (mgr.14176) 771 : cluster [DBG] pgmap v552: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:37 smithi156 bash[29655]: cluster 2023-11-17T05:50:36.101870+0000 mgr.smithi006.odexej (mgr.14176) 772 : cluster [DBG] pgmap v553: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:39 smithi156 bash[29655]: cluster 2023-11-17T05:50:38.102826+0000 mgr.smithi006.odexej (mgr.14176) 773 : cluster [DBG] pgmap v554: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:41 smithi156 bash[29655]: cluster 2023-11-17T05:50:40.104270+0000 mgr.smithi006.odexej (mgr.14176) 774 : cluster [DBG] pgmap v555: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.349 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:43 smithi156 bash[29655]: cluster 2023-11-17T05:50:42.105294+0000 mgr.smithi006.odexej (mgr.14176) 775 : cluster [DBG] pgmap v556: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:45 smithi156 bash[29655]: cluster 2023-11-17T05:50:44.106429+0000 mgr.smithi006.odexej (mgr.14176) 776 : cluster [DBG] pgmap v557: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:47 smithi156 bash[29655]: cluster 2023-11-17T05:50:46.107775+0000 mgr.smithi006.odexej (mgr.14176) 777 : cluster [DBG] pgmap v558: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.075462+0000 mon.smithi006 (mon.0) 851 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.400415+0000 mon.smithi006 (mon.0) 852 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.402151+0000 mon.smithi006 (mon.0) 853 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.410945+0000 mon.smithi006 (mon.0) 854 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.350 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.417601+0000 mon.smithi006 (mon.0) 855 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:48 smithi156 bash[29655]: audit 2023-11-17T05:50:48.435590+0000 mon.smithi006 (mon.0) 856 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:49 smithi156 bash[29655]: cluster 2023-11-17T05:50:48.108792+0000 mgr.smithi006.odexej (mgr.14176) 778 : cluster [DBG] pgmap v559: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:49 smithi156 bash[29655]: cluster 2023-11-17T05:50:48.412024+0000 mgr.smithi006.odexej (mgr.14176) 779 : cluster [DBG] pgmap v560: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:49 smithi156 bash[29655]: cephadm 2023-11-17T05:50:48.440555+0000 mgr.smithi006.odexej (mgr.14176) 780 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.kzugpe on smithi156 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:49 smithi156 bash[29655]: cluster 2023-11-17T05:50:48.690931+0000 mon.smithi006 (mon.0) 857 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:51:19.351 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:49 smithi156 bash[29655]: cluster 2023-11-17T05:50:48.691013+0000 mon.smithi006 (mon.0) 858 : cluster [INF] Cluster is now healthy 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:50 smithi156 bash[29655]: audit 2023-11-17T05:50:49.717191+0000 mon.smithi006 (mon.0) 859 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:51 smithi156 bash[29655]: cluster 2023-11-17T05:50:50.413113+0000 mgr.smithi006.odexej (mgr.14176) 781 : cluster [DBG] pgmap v561: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:52 smithi156 bash[29655]: audit 2023-11-17T05:50:52.201288+0000 mon.smithi006 (mon.0) 860 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.kzugpe"}]: dispatch 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: cephadm 2023-11-17T05:50:52.199986+0000 mgr.smithi006.odexej (mgr.14176) 782 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.352 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.353 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:51:19.354 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:19.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: cephadm 2023-11-17T05:50:52.200525+0000 mgr.smithi006.odexej (mgr.14176) 783 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: cephadm 2023-11-17T05:50:52.203134+0000 mgr.smithi006.odexej (mgr.14176) 784 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.356 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.357 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: cephadm 2023-11-17T05:50:52.208043+0000 mgr.smithi006.odexej (mgr.14176) 785 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.guegsz on smithi006 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:53 smithi156 bash[29655]: cluster 2023-11-17T05:50:52.413650+0000 mgr.smithi006.odexej (mgr.14176) 786 : cluster [DBG] pgmap v562: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T05:51:19.358 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:55 smithi156 bash[29655]: cluster 2023-11-17T05:50:54.414730+0000 mgr.smithi006.odexej (mgr.14176) 787 : cluster [DBG] pgmap v563: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cephadm 2023-11-17T05:50:55.968289+0000 mgr.smithi006.odexej (mgr.14176) 788 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.359 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:51:19.360 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.361 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cephadm 2023-11-17T05:50:55.968737+0000 mgr.smithi006.odexej (mgr.14176) 789 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.362 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: audit 2023-11-17T05:50:55.969335+0000 mon.smithi006 (mon.0) 861 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.guegsz"}]: dispatch 2023-11-17T05:51:19.481 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cephadm 2023-11-17T05:50:55.970723+0000 mgr.smithi006.odexej (mgr.14176) 790 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.482 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.482 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:19.482 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.482 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:51:19.488 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cephadm 2023-11-17T05:50:55.974197+0000 mgr.smithi006.odexej (mgr.14176) 791 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:51:19.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cephadm 2023-11-17T05:50:55.977792+0000 mgr.smithi006.odexej (mgr.14176) 792 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:51:19.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:56 smithi156 bash[29655]: cluster 2023-11-17T05:50:55.979288+0000 mgr.smithi006.odexej (mgr.14176) 793 : cluster [DBG] pgmap v564: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:51:19.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:57 smithi156 bash[29655]: cluster 2023-11-17T05:50:56.746318+0000 mon.smithi006 (mon.0) 862 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:51:19.625 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:50:58 smithi156 bash[29655]: cluster 2023-11-17T05:50:57.980355+0000 mgr.smithi006.odexej (mgr.14176) 794 : cluster [DBG] pgmap v565: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:01 smithi156 bash[29655]: cluster 2023-11-17T05:50:59.981775+0000 mgr.smithi006.odexej (mgr.14176) 795 : cluster [DBG] pgmap v566: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:01 smithi156 bash[29655]: audit 2023-11-17T05:51:00.508223+0000 mon.smithi006 (mon.0) 863 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:02 smithi156 bash[29655]: cluster 2023-11-17T05:51:01.982860+0000 mgr.smithi006.odexej (mgr.14176) 796 : cluster [DBG] pgmap v567: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:05 smithi156 bash[29655]: cluster 2023-11-17T05:51:03.984062+0000 mgr.smithi006.odexej (mgr.14176) 797 : cluster [DBG] pgmap v568: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:05 smithi156 bash[29655]: audit 2023-11-17T05:51:04.718284+0000 mon.smithi006 (mon.0) 864 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:07 smithi156 bash[29655]: cluster 2023-11-17T05:51:05.985452+0000 mgr.smithi006.odexej (mgr.14176) 798 : cluster [DBG] pgmap v569: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:19.627 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:09 smithi156 bash[29655]: cluster 2023-11-17T05:51:07.986442+0000 mgr.smithi006.odexej (mgr.14176) 799 : cluster [DBG] pgmap v570: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.627 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:11 smithi156 bash[29655]: cluster 2023-11-17T05:51:09.987888+0000 mgr.smithi006.odexej (mgr.14176) 800 : cluster [DBG] pgmap v571: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.627 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:12 smithi156 bash[29655]: cluster 2023-11-17T05:51:11.988905+0000 mgr.smithi006.odexej (mgr.14176) 801 : cluster [DBG] pgmap v572: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.627 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:15 smithi156 bash[29655]: cluster 2023-11-17T05:51:13.990100+0000 mgr.smithi006.odexej (mgr.14176) 802 : cluster [DBG] pgmap v573: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.627 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:17 smithi156 bash[29655]: cluster 2023-11-17T05:51:15.991444+0000 mgr.smithi006.odexej (mgr.14176) 803 : cluster [DBG] pgmap v574: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:03 smithi006 bash[26022]: cluster 2023-11-17T05:50:02.080924+0000 mgr.smithi006.odexej (mgr.14176) 755 : cluster [DBG] pgmap v536: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:05 smithi006 bash[26022]: cluster 2023-11-17T05:50:04.082116+0000 mgr.smithi006.odexej (mgr.14176) 756 : cluster [DBG] pgmap v537: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:07 smithi006 bash[26022]: cluster 2023-11-17T05:50:06.083498+0000 mgr.smithi006.odexej (mgr.14176) 757 : cluster [DBG] pgmap v538: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:09 smithi006 bash[26022]: cluster 2023-11-17T05:50:08.084525+0000 mgr.smithi006.odexej (mgr.14176) 758 : cluster [DBG] pgmap v539: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:11 smithi006 bash[26022]: cluster 2023-11-17T05:50:10.085955+0000 mgr.smithi006.odexej (mgr.14176) 759 : cluster [DBG] pgmap v540: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.628 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:13 smithi006 bash[26022]: cluster 2023-11-17T05:50:12.087008+0000 mgr.smithi006.odexej (mgr.14176) 760 : cluster [DBG] pgmap v541: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.974 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:15 smithi006 bash[26022]: cluster 2023-11-17T05:50:14.088311+0000 mgr.smithi006.odexej (mgr.14176) 761 : cluster [DBG] pgmap v542: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:17 smithi006 bash[26022]: cluster 2023-11-17T05:50:16.089687+0000 mgr.smithi006.odexej (mgr.14176) 762 : cluster [DBG] pgmap v543: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:19 smithi006 bash[26022]: cluster 2023-11-17T05:50:18.090674+0000 mgr.smithi006.odexej (mgr.14176) 763 : cluster [DBG] pgmap v544: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:21 smithi006 bash[26022]: cluster 2023-11-17T05:50:20.092063+0000 mgr.smithi006.odexej (mgr.14176) 764 : cluster [DBG] pgmap v545: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:23 smithi006 bash[26022]: cluster 2023-11-17T05:50:22.093060+0000 mgr.smithi006.odexej (mgr.14176) 765 : cluster [DBG] pgmap v546: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:25 smithi006 bash[26022]: cluster 2023-11-17T05:50:24.094244+0000 mgr.smithi006.odexej (mgr.14176) 766 : cluster [DBG] pgmap v547: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:27 smithi006 bash[26022]: cluster 2023-11-17T05:50:26.095744+0000 mgr.smithi006.odexej (mgr.14176) 767 : cluster [DBG] pgmap v548: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:29 smithi006 bash[26022]: cluster 2023-11-17T05:50:28.096903+0000 mgr.smithi006.odexej (mgr.14176) 768 : cluster [DBG] pgmap v549: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:31 smithi006 bash[26022]: cluster 2023-11-17T05:50:30.098339+0000 mgr.smithi006.odexej (mgr.14176) 769 : cluster [DBG] pgmap v550: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:33 smithi006 bash[26022]: cluster 2023-11-17T05:50:32.099377+0000 mgr.smithi006.odexej (mgr.14176) 770 : cluster [DBG] pgmap v551: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:35 smithi006 bash[26022]: cluster 2023-11-17T05:50:34.100516+0000 mgr.smithi006.odexej (mgr.14176) 771 : cluster [DBG] pgmap v552: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:37 smithi006 bash[26022]: cluster 2023-11-17T05:50:36.101870+0000 mgr.smithi006.odexej (mgr.14176) 772 : cluster [DBG] pgmap v553: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:39 smithi006 bash[26022]: cluster 2023-11-17T05:50:38.102826+0000 mgr.smithi006.odexej (mgr.14176) 773 : cluster [DBG] pgmap v554: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.975 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:41 smithi006 bash[26022]: cluster 2023-11-17T05:50:40.104270+0000 mgr.smithi006.odexej (mgr.14176) 774 : cluster [DBG] pgmap v555: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:43 smithi006 bash[26022]: cluster 2023-11-17T05:50:42.105294+0000 mgr.smithi006.odexej (mgr.14176) 775 : cluster [DBG] pgmap v556: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:45 smithi006 bash[26022]: cluster 2023-11-17T05:50:44.106429+0000 mgr.smithi006.odexej (mgr.14176) 776 : cluster [DBG] pgmap v557: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:47 smithi006 bash[26022]: cluster 2023-11-17T05:50:46.107775+0000 mgr.smithi006.odexej (mgr.14176) 777 : cluster [DBG] pgmap v558: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.075462+0000 mon.smithi006 (mon.0) 851 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.400415+0000 mon.smithi006 (mon.0) 852 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.402151+0000 mon.smithi006 (mon.0) 853 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.410945+0000 mon.smithi006 (mon.0) 854 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.417601+0000 mon.smithi006 (mon.0) 855 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:48 smithi006 bash[26022]: audit 2023-11-17T05:50:48.435590+0000 mon.smithi006 (mon.0) 856 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:49 smithi006 bash[26022]: cluster 2023-11-17T05:50:48.108792+0000 mgr.smithi006.odexej (mgr.14176) 778 : cluster [DBG] pgmap v559: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:49 smithi006 bash[26022]: cluster 2023-11-17T05:50:48.412024+0000 mgr.smithi006.odexej (mgr.14176) 779 : cluster [DBG] pgmap v560: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:49 smithi006 bash[26022]: cephadm 2023-11-17T05:50:48.440555+0000 mgr.smithi006.odexej (mgr.14176) 780 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.kzugpe on smithi156 2023-11-17T05:51:19.976 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:49 smithi006 bash[26022]: cluster 2023-11-17T05:50:48.690931+0000 mon.smithi006 (mon.0) 857 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:49 smithi006 bash[26022]: cluster 2023-11-17T05:50:48.691013+0000 mon.smithi006 (mon.0) 858 : cluster [INF] Cluster is now healthy 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:50 smithi006 bash[26022]: audit 2023-11-17T05:50:49.717191+0000 mon.smithi006 (mon.0) 859 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:51 smithi006 bash[26022]: cluster 2023-11-17T05:50:50.413113+0000 mgr.smithi006.odexej (mgr.14176) 781 : cluster [DBG] pgmap v561: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:52 smithi006 bash[26022]: audit 2023-11-17T05:50:52.201288+0000 mon.smithi006 (mon.0) 860 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.kzugpe"}]: dispatch 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: cephadm 2023-11-17T05:50:52.199986+0000 mgr.smithi006.odexej (mgr.14176) 782 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:19.977 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.978 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:19.978 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:19.978 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:19.978 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:19.978 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:51:20.031 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:51:20.031 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:51:20.031 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:51:20.031 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:51:20.031 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: cephadm 2023-11-17T05:50:52.200525+0000 mgr.smithi006.odexej (mgr.14176) 783 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:20.032 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: cephadm 2023-11-17T05:50:52.203134+0000 mgr.smithi006.odexej (mgr.14176) 784 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kzugpe ... 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: cephadm 2023-11-17T05:50:52.208043+0000 mgr.smithi006.odexej (mgr.14176) 785 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.guegsz on smithi006 2023-11-17T05:51:20.033 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:53 smithi006 bash[26022]: cluster 2023-11-17T05:50:52.413650+0000 mgr.smithi006.odexej (mgr.14176) 786 : cluster [DBG] pgmap v562: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:55 smithi006 bash[26022]: cluster 2023-11-17T05:50:54.414730+0000 mgr.smithi006.odexej (mgr.14176) 787 : cluster [DBG] pgmap v563: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cephadm 2023-11-17T05:50:55.968289+0000 mgr.smithi006.odexej (mgr.14176) 788 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:20.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:20.035 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.035 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cephadm 2023-11-17T05:50:55.968737+0000 mgr.smithi006.odexej (mgr.14176) 789 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: audit 2023-11-17T05:50:55.969335+0000 mon.smithi006 (mon.0) 861 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.guegsz"}]: dispatch 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cephadm 2023-11-17T05:50:55.970723+0000 mgr.smithi006.odexej (mgr.14176) 790 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.guegsz ... 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cephadm 2023-11-17T05:50:55.974197+0000 mgr.smithi006.odexej (mgr.14176) 791 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cephadm 2023-11-17T05:50:55.977792+0000 mgr.smithi006.odexej (mgr.14176) 792 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:56 smithi006 bash[26022]: cluster 2023-11-17T05:50:55.979288+0000 mgr.smithi006.odexej (mgr.14176) 793 : cluster [DBG] pgmap v564: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:51:20.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:57 smithi006 bash[26022]: cluster 2023-11-17T05:50:56.746318+0000 mon.smithi006 (mon.0) 862 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:50:58 smithi006 bash[26022]: cluster 2023-11-17T05:50:57.980355+0000 mgr.smithi006.odexej (mgr.14176) 794 : cluster [DBG] pgmap v565: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:01 smithi006 bash[26022]: cluster 2023-11-17T05:50:59.981775+0000 mgr.smithi006.odexej (mgr.14176) 795 : cluster [DBG] pgmap v566: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:01 smithi006 bash[26022]: audit 2023-11-17T05:51:00.508223+0000 mon.smithi006 (mon.0) 863 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:02 smithi006 bash[26022]: cluster 2023-11-17T05:51:01.982860+0000 mgr.smithi006.odexej (mgr.14176) 796 : cluster [DBG] pgmap v567: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:05 smithi006 bash[26022]: cluster 2023-11-17T05:51:03.984062+0000 mgr.smithi006.odexej (mgr.14176) 797 : cluster [DBG] pgmap v568: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:05 smithi006 bash[26022]: audit 2023-11-17T05:51:04.718284+0000 mon.smithi006 (mon.0) 864 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:07 smithi006 bash[26022]: cluster 2023-11-17T05:51:05.985452+0000 mgr.smithi006.odexej (mgr.14176) 798 : cluster [DBG] pgmap v569: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:09 smithi006 bash[26022]: cluster 2023-11-17T05:51:07.986442+0000 mgr.smithi006.odexej (mgr.14176) 799 : cluster [DBG] pgmap v570: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:11 smithi006 bash[26022]: cluster 2023-11-17T05:51:09.987888+0000 mgr.smithi006.odexej (mgr.14176) 800 : cluster [DBG] pgmap v571: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:20.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:12 smithi006 bash[26022]: cluster 2023-11-17T05:51:11.988905+0000 mgr.smithi006.odexej (mgr.14176) 801 : cluster [DBG] pgmap v572: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:20.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:15 smithi006 bash[26022]: cluster 2023-11-17T05:51:13.990100+0000 mgr.smithi006.odexej (mgr.14176) 802 : cluster [DBG] pgmap v573: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:20.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:17 smithi006 bash[26022]: cluster 2023-11-17T05:51:15.991444+0000 mgr.smithi006.odexej (mgr.14176) 803 : cluster [DBG] pgmap v574: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:20.117 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:19 smithi006 bash[26022]: cluster 2023-11-17T05:51:17.992520+0000 mgr.smithi006.odexej (mgr.14176) 804 : cluster [DBG] pgmap v575: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:20.117 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:19 smithi156 bash[29655]: cluster 2023-11-17T05:51:17.992520+0000 mgr.smithi006.odexej (mgr.14176) 804 : cluster [DBG] pgmap v575: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:21.402 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:21 smithi006 bash[26022]: cluster 2023-11-17T05:51:19.993861+0000 mgr.smithi006.odexej (mgr.14176) 805 : cluster [DBG] pgmap v576: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:21.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:21 smithi156 bash[29655]: cluster 2023-11-17T05:51:19.993861+0000 mgr.smithi006.odexej (mgr.14176) 805 : cluster [DBG] pgmap v576: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:22.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:22 smithi006 bash[26022]: cluster 2023-11-17T05:51:21.994885+0000 mgr.smithi006.odexej (mgr.14176) 806 : cluster [DBG] pgmap v577: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:22.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:22 smithi156 bash[29655]: cluster 2023-11-17T05:51:21.994885+0000 mgr.smithi006.odexej (mgr.14176) 806 : cluster [DBG] pgmap v577: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:23.062 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:23.063 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:23.658 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:23.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:23 smithi006 bash[26022]: audit 2023-11-17T05:51:23.045283+0000 mgr.smithi006.odexej (mgr.14176) 807 : audit [DBG] from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:23.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:23 smithi156 bash[29655]: audit 2023-11-17T05:51:23.045283+0000 mgr.smithi006.odexej (mgr.14176) 807 : audit [DBG] from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:24.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:24.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:24 smithi006 bash[26022]: cluster 2023-11-17T05:51:23.996066+0000 mgr.smithi006.odexej (mgr.14176) 808 : cluster [DBG] pgmap v578: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:24.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:24 smithi156 bash[29655]: cluster 2023-11-17T05:51:23.996066+0000 mgr.smithi006.odexej (mgr.14176) 808 : cluster [DBG] pgmap v578: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:27.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:27 smithi006 bash[26022]: cluster 2023-11-17T05:51:25.997608+0000 mgr.smithi006.odexej (mgr.14176) 809 : cluster [DBG] pgmap v579: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:27.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:27 smithi156 bash[29655]: cluster 2023-11-17T05:51:25.997608+0000 mgr.smithi006.odexej (mgr.14176) 809 : cluster [DBG] pgmap v579: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:27.658 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:28.322 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:29.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:29.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:29 smithi006 bash[26022]: audit 2023-11-17T05:51:27.637915+0000 mgr.smithi006.odexej (mgr.14176) 810 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:29.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:29 smithi006 bash[26022]: cluster 2023-11-17T05:51:27.998717+0000 mgr.smithi006.odexej (mgr.14176) 811 : cluster [DBG] pgmap v580: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:29 smithi156 bash[29655]: audit 2023-11-17T05:51:27.637915+0000 mgr.smithi006.odexej (mgr.14176) 810 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:29 smithi156 bash[29655]: cluster 2023-11-17T05:51:27.998717+0000 mgr.smithi006.odexej (mgr.14176) 811 : cluster [DBG] pgmap v580: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:31.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:31 smithi006 bash[26022]: cluster 2023-11-17T05:51:30.000177+0000 mgr.smithi006.odexej (mgr.14176) 812 : cluster [DBG] pgmap v581: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:31 smithi156 bash[29655]: cluster 2023-11-17T05:51:30.000177+0000 mgr.smithi006.odexej (mgr.14176) 812 : cluster [DBG] pgmap v581: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:32.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:32.474 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:32 smithi006 bash[26022]: cluster 2023-11-17T05:51:32.001050+0000 mgr.smithi006.odexej (mgr.14176) 813 : cluster [DBG] pgmap v582: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:32.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:32 smithi156 bash[29655]: cluster 2023-11-17T05:51:32.001050+0000 mgr.smithi006.odexej (mgr.14176) 813 : cluster [DBG] pgmap v582: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:33.110 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:33.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:33 smithi006 bash[26022]: audit 2023-11-17T05:51:32.453538+0000 mgr.smithi006.odexej (mgr.14176) 814 : audit [DBG] from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:33.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:33 smithi156 bash[29655]: audit 2023-11-17T05:51:32.453538+0000 mgr.smithi006.odexej (mgr.14176) 814 : audit [DBG] from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:34.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:34.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:34 smithi006 bash[26022]: cluster 2023-11-17T05:51:34.002295+0000 mgr.smithi006.odexej (mgr.14176) 815 : cluster [DBG] pgmap v583: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:34.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:34 smithi156 bash[29655]: cluster 2023-11-17T05:51:34.002295+0000 mgr.smithi006.odexej (mgr.14176) 815 : cluster [DBG] pgmap v583: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:37.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:37 smithi006 bash[26022]: cluster 2023-11-17T05:51:36.004004+0000 mgr.smithi006.odexej (mgr.14176) 816 : cluster [DBG] pgmap v584: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:37.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:37.279 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:37.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:37 smithi156 bash[29655]: cluster 2023-11-17T05:51:36.004004+0000 mgr.smithi006.odexej (mgr.14176) 816 : cluster [DBG] pgmap v584: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:37.921 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:38.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:39.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:39 smithi006 bash[26022]: audit 2023-11-17T05:51:37.262033+0000 mgr.smithi006.odexej (mgr.14176) 817 : audit [DBG] from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:39.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:39 smithi006 bash[26022]: cluster 2023-11-17T05:51:38.005106+0000 mgr.smithi006.odexej (mgr.14176) 818 : cluster [DBG] pgmap v585: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:39.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:39 smithi156 bash[29655]: audit 2023-11-17T05:51:37.262033+0000 mgr.smithi006.odexej (mgr.14176) 817 : audit [DBG] from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:39.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:39 smithi156 bash[29655]: cluster 2023-11-17T05:51:38.005106+0000 mgr.smithi006.odexej (mgr.14176) 818 : cluster [DBG] pgmap v585: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:40.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:40 smithi006 bash[26022]: cluster 2023-11-17T05:51:40.006873+0000 mgr.smithi006.odexej (mgr.14176) 819 : cluster [DBG] pgmap v586: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:40.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:40 smithi156 bash[29655]: cluster 2023-11-17T05:51:40.006873+0000 mgr.smithi006.odexej (mgr.14176) 819 : cluster [DBG] pgmap v586: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:41.937 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:41.938 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:42.535 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:42.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:42 smithi006 bash[26022]: audit 2023-11-17T05:51:41.925739+0000 mgr.smithi006.odexej (mgr.14176) 820 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:42.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:42 smithi006 bash[26022]: cluster 2023-11-17T05:51:42.008028+0000 mgr.smithi006.odexej (mgr.14176) 821 : cluster [DBG] pgmap v587: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:42.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:42 smithi156 bash[29655]: audit 2023-11-17T05:51:41.925739+0000 mgr.smithi006.odexej (mgr.14176) 820 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:42.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:42 smithi156 bash[29655]: cluster 2023-11-17T05:51:42.008028+0000 mgr.smithi006.odexej (mgr.14176) 821 : cluster [DBG] pgmap v587: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:43.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:45.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:45 smithi006 bash[26022]: cluster 2023-11-17T05:51:44.009365+0000 mgr.smithi006.odexej (mgr.14176) 822 : cluster [DBG] pgmap v588: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:45.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:45 smithi156 bash[29655]: cluster 2023-11-17T05:51:44.009365+0000 mgr.smithi006.odexej (mgr.14176) 822 : cluster [DBG] pgmap v588: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:46.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:46.704 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:47.306 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:47.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:47 smithi006 bash[26022]: cluster 2023-11-17T05:51:46.010951+0000 mgr.smithi006.odexej (mgr.14176) 823 : cluster [DBG] pgmap v589: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:47.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:47 smithi156 bash[29655]: cluster 2023-11-17T05:51:46.010951+0000 mgr.smithi006.odexej (mgr.14176) 823 : cluster [DBG] pgmap v589: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:48.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:48.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:48 smithi006 bash[26022]: audit 2023-11-17T05:51:46.684494+0000 mgr.smithi006.odexej (mgr.14176) 824 : audit [DBG] from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:48 smithi156 bash[29655]: audit 2023-11-17T05:51:46.684494+0000 mgr.smithi006.odexej (mgr.14176) 824 : audit [DBG] from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:49.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:49 smithi006 bash[26022]: cluster 2023-11-17T05:51:48.012091+0000 mgr.smithi006.odexej (mgr.14176) 825 : cluster [DBG] pgmap v590: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:49.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:49 smithi156 bash[29655]: cluster 2023-11-17T05:51:48.012091+0000 mgr.smithi006.odexej (mgr.14176) 825 : cluster [DBG] pgmap v590: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:50.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:50 smithi006 bash[26022]: cluster 2023-11-17T05:51:50.013623+0000 mgr.smithi006.odexej (mgr.14176) 826 : cluster [DBG] pgmap v591: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:50.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:50 smithi156 bash[29655]: cluster 2023-11-17T05:51:50.013623+0000 mgr.smithi006.odexej (mgr.14176) 826 : cluster [DBG] pgmap v591: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:51.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:51.307 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:51.890 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:52.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:52.906 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:52 smithi006 bash[26022]: audit 2023-11-17T05:51:51.289044+0000 mgr.smithi006.odexej (mgr.14176) 827 : audit [DBG] from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:52.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:52 smithi006 bash[26022]: cluster 2023-11-17T05:51:52.014863+0000 mgr.smithi006.odexej (mgr.14176) 828 : cluster [DBG] pgmap v592: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:52.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:52 smithi156 bash[29655]: audit 2023-11-17T05:51:51.289044+0000 mgr.smithi006.odexej (mgr.14176) 827 : audit [DBG] from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:52.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:52 smithi156 bash[29655]: cluster 2023-11-17T05:51:52.014863+0000 mgr.smithi006.odexej (mgr.14176) 828 : cluster [DBG] pgmap v592: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:55.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:55 smithi006 bash[26022]: cluster 2023-11-17T05:51:54.015699+0000 mgr.smithi006.odexej (mgr.14176) 829 : cluster [DBG] pgmap v593: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:55.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:55 smithi156 bash[29655]: cluster 2023-11-17T05:51:54.015699+0000 mgr.smithi006.odexej (mgr.14176) 829 : cluster [DBG] pgmap v593: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:51:56.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:51:56.182 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:51:56.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:56 smithi006 bash[26022]: audit 2023-11-17T05:51:55.982793+0000 mon.smithi006 (mon.0) 865 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:51:56.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:56 smithi006 bash[26022]: cluster 2023-11-17T05:51:56.016673+0000 mgr.smithi006.odexej (mgr.14176) 830 : cluster [DBG] pgmap v594: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:56 smithi156 bash[29655]: audit 2023-11-17T05:51:55.982793+0000 mon.smithi006 (mon.0) 865 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:51:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:56 smithi156 bash[29655]: cluster 2023-11-17T05:51:56.016673+0000 mgr.smithi006.odexej (mgr.14176) 830 : cluster [DBG] pgmap v594: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:56.804 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:51:57.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:51:57.817 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:57 smithi006 bash[26022]: audit 2023-11-17T05:51:56.165689+0000 mgr.smithi006.odexej (mgr.14176) 831 : audit [DBG] from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:57.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:57 smithi156 bash[29655]: audit 2023-11-17T05:51:56.165689+0000 mgr.smithi006.odexej (mgr.14176) 831 : audit [DBG] from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:51:58.827 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:51:58 smithi006 bash[26022]: cluster 2023-11-17T05:51:58.017400+0000 mgr.smithi006.odexej (mgr.14176) 832 : cluster [DBG] pgmap v595: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:51:58.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:51:58 smithi156 bash[29655]: cluster 2023-11-17T05:51:58.017400+0000 mgr.smithi006.odexej (mgr.14176) 832 : cluster [DBG] pgmap v595: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:00.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:00.784 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:41:43.755692Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:41:42.972102Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:41:43.756217Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:41:42.974026Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:41:42.972499Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:41:42.972767Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:41:43.758286Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:41:42.973012Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:41:42.973234Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:41:43.756727Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:01.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:01 smithi006 bash[26022]: cluster 2023-11-17T05:52:00.018766+0000 mgr.smithi006.odexej (mgr.14176) 833 : cluster [DBG] pgmap v596: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:01.366 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:01.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:01 smithi156 bash[29655]: cluster 2023-11-17T05:52:00.018766+0000 mgr.smithi006.odexej (mgr.14176) 833 : cluster [DBG] pgmap v596: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:02.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:02.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:02 smithi006 bash[26022]: audit 2023-11-17T05:52:00.772366+0000 mgr.smithi006.odexej (mgr.14176) 834 : audit [DBG] from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:02.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:02 smithi006 bash[26022]: audit 2023-11-17T05:52:01.681702+0000 mon.smithi006 (mon.0) 866 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:02.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:02 smithi156 bash[29655]: audit 2023-11-17T05:52:00.772366+0000 mgr.smithi006.odexej (mgr.14176) 834 : audit [DBG] from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:02.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:02 smithi156 bash[29655]: audit 2023-11-17T05:52:01.681702+0000 mon.smithi006 (mon.0) 866 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: cluster 2023-11-17T05:52:02.019727+0000 mgr.smithi006.odexej (mgr.14176) 835 : cluster [DBG] pgmap v597: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.429562+0000 mon.smithi006 (mon.0) 867 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.731601+0000 mon.smithi006 (mon.0) 868 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.733204+0000 mon.smithi006 (mon.0) 869 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.741426+0000 mon.smithi006 (mon.0) 870 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: cluster 2023-11-17T05:52:02.742729+0000 mgr.smithi006.odexej (mgr.14176) 836 : cluster [DBG] pgmap v598: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.750926+0000 mon.smithi006 (mon.0) 871 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: audit 2023-11-17T05:52:02.765232+0000 mon.smithi006 (mon.0) 872 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:03 smithi006 bash[26022]: cephadm 2023-11-17T05:52:02.768361+0000 mgr.smithi006.odexej (mgr.14176) 837 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.drplam on smithi156 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: cluster 2023-11-17T05:52:02.019727+0000 mgr.smithi006.odexej (mgr.14176) 835 : cluster [DBG] pgmap v597: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.429562+0000 mon.smithi006 (mon.0) 867 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.731601+0000 mon.smithi006 (mon.0) 868 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.733204+0000 mon.smithi006 (mon.0) 869 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.741426+0000 mon.smithi006 (mon.0) 870 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: cluster 2023-11-17T05:52:02.742729+0000 mgr.smithi006.odexej (mgr.14176) 836 : cluster [DBG] pgmap v598: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.750926+0000 mon.smithi006 (mon.0) 871 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: audit 2023-11-17T05:52:02.765232+0000 mon.smithi006 (mon.0) 872 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:03 smithi156 bash[29655]: cephadm 2023-11-17T05:52:02.768361+0000 mgr.smithi006.odexej (mgr.14176) 837 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.drplam on smithi156 2023-11-17T05:52:04.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:04 smithi006 bash[26022]: cluster 2023-11-17T05:52:03.738477+0000 mon.smithi006 (mon.0) 873 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:52:04.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:04 smithi006 bash[26022]: cluster 2023-11-17T05:52:03.738537+0000 mon.smithi006 (mon.0) 874 : cluster [INF] Cluster is now healthy 2023-11-17T05:52:04.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:04 smithi156 bash[29655]: cluster 2023-11-17T05:52:03.738477+0000 mon.smithi006 (mon.0) 873 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:52:04.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:04 smithi156 bash[29655]: cluster 2023-11-17T05:52:03.738537+0000 mon.smithi006 (mon.0) 874 : cluster [INF] Cluster is now healthy 2023-11-17T05:52:05.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:05.526 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:49:48.071379Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zydgom on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:52.202901Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kzugpe on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kzugpe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kzugpe\nDeploy daemon haproxy.nfs.foo.smithi156.kzugpe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.970522Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.guegsz on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-guegsz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.guegsz\nDeploy daemon haproxy.nfs.foo.smithi006.guegsz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:50:55.974013Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.plymvv on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:05.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:05 smithi006 bash[26022]: audit 2023-11-17T05:52:04.719776+0000 mon.smithi006 (mon.0) 875 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:05.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:05 smithi006 bash[26022]: cluster 2023-11-17T05:52:04.744100+0000 mgr.smithi006.odexej (mgr.14176) 838 : cluster [DBG] pgmap v599: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:06.170 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:06.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:05 smithi156 bash[29655]: audit 2023-11-17T05:52:04.719776+0000 mon.smithi006 (mon.0) 875 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:06.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:05 smithi156 bash[29655]: cluster 2023-11-17T05:52:04.744100+0000 mgr.smithi006.odexej (mgr.14176) 838 : cluster [DBG] pgmap v599: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:06.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:06 smithi006 bash[26022]: audit 2023-11-17T05:52:05.506560+0000 mgr.smithi006.odexej (mgr.14176) 839 : audit [DBG] from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:06.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:06 smithi006 bash[26022]: audit 2023-11-17T05:52:06.523167+0000 mon.smithi006 (mon.0) 876 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.drplam"}]: dispatch 2023-11-17T05:52:07.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:07.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:06 smithi156 bash[29655]: audit 2023-11-17T05:52:05.506560+0000 mgr.smithi006.odexej (mgr.14176) 839 : audit [DBG] from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:07.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:06 smithi156 bash[29655]: audit 2023-11-17T05:52:06.523167+0000 mon.smithi006 (mon.0) 876 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.drplam"}]: dispatch 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: cephadm 2023-11-17T05:52:06.521947+0000 mgr.smithi006.odexej (mgr.14176) 840 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:52:08.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: cephadm 2023-11-17T05:52:06.522466+0000 mgr.smithi006.odexej (mgr.14176) 841 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: cephadm 2023-11-17T05:52:06.524905+0000 mgr.smithi006.odexej (mgr.14176) 842 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: cephadm 2023-11-17T05:52:06.530083+0000 mgr.smithi006.odexej (mgr.14176) 843 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.pomqkk on smithi006 2023-11-17T05:52:08.174 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:07 smithi006 bash[26022]: cluster 2023-11-17T05:52:06.744849+0000 mgr.smithi006.odexej (mgr.14176) 844 : cluster [DBG] pgmap v600: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:08.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: cephadm 2023-11-17T05:52:06.521947+0000 mgr.smithi006.odexej (mgr.14176) 840 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:52:08.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: cephadm 2023-11-17T05:52:06.522466+0000 mgr.smithi006.odexej (mgr.14176) 841 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: cephadm 2023-11-17T05:52:06.524905+0000 mgr.smithi006.odexej (mgr.14176) 842 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.drplam ... 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:08.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:08.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:08.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:08.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: cephadm 2023-11-17T05:52:06.530083+0000 mgr.smithi006.odexej (mgr.14176) 843 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.pomqkk on smithi006 2023-11-17T05:52:08.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:07 smithi156 bash[29655]: cluster 2023-11-17T05:52:06.744849+0000 mgr.smithi006.odexej (mgr.14176) 844 : cluster [DBG] pgmap v600: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:10.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:09 smithi006 bash[26022]: cluster 2023-11-17T05:52:08.745748+0000 mgr.smithi006.odexej (mgr.14176) 845 : cluster [DBG] pgmap v601: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:10.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:09 smithi156 bash[29655]: cluster 2023-11-17T05:52:08.745748+0000 mgr.smithi006.odexej (mgr.14176) 845 : cluster [DBG] pgmap v601: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 B/s rd, 0 op/s 2023-11-17T05:52:10.389 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:10.390 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:11.030 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:11.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:10 smithi006 bash[26022]: audit 2023-11-17T05:52:10.279122+0000 mon.smithi006 (mon.0) 877 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.pomqkk"}]: dispatch 2023-11-17T05:52:11.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:10 smithi006 bash[26022]: audit 2023-11-17T05:52:10.558870+0000 mon.smithi006 (mon.0) 878 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:11.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:10 smithi156 bash[29655]: audit 2023-11-17T05:52:10.279122+0000 mon.smithi006 (mon.0) 877 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.pomqkk"}]: dispatch 2023-11-17T05:52:11.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:10 smithi156 bash[29655]: audit 2023-11-17T05:52:10.558870+0000 mon.smithi006 (mon.0) 878 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:12.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cephadm 2023-11-17T05:52:10.277966+0000 mgr.smithi006.odexej (mgr.14176) 846 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cephadm 2023-11-17T05:52:10.278559+0000 mgr.smithi006.odexej (mgr.14176) 847 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cephadm 2023-11-17T05:52:10.280371+0000 mgr.smithi006.odexej (mgr.14176) 848 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cephadm 2023-11-17T05:52:10.284581+0000 mgr.smithi006.odexej (mgr.14176) 849 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cephadm 2023-11-17T05:52:10.288530+0000 mgr.smithi006.odexej (mgr.14176) 850 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cluster 2023-11-17T05:52:10.290230+0000 mgr.smithi006.odexej (mgr.14176) 851 : cluster [DBG] pgmap v602: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:52:12.172 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: audit 2023-11-17T05:52:10.370698+0000 mgr.smithi006.odexej (mgr.14176) 852 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:12.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:11 smithi006 bash[26022]: cluster 2023-11-17T05:52:10.803272+0000 mon.smithi006 (mon.0) 879 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:52:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cephadm 2023-11-17T05:52:10.277966+0000 mgr.smithi006.odexej (mgr.14176) 846 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:52:12.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cephadm 2023-11-17T05:52:10.278559+0000 mgr.smithi006.odexej (mgr.14176) 847 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cephadm 2023-11-17T05:52:10.280371+0000 mgr.smithi006.odexej (mgr.14176) 848 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:52:12.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.pomqkk ... 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cephadm 2023-11-17T05:52:10.284581+0000 mgr.smithi006.odexej (mgr.14176) 849 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cephadm 2023-11-17T05:52:10.288530+0000 mgr.smithi006.odexej (mgr.14176) 850 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cluster 2023-11-17T05:52:10.290230+0000 mgr.smithi006.odexej (mgr.14176) 851 : cluster [DBG] pgmap v602: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: audit 2023-11-17T05:52:10.370698+0000 mgr.smithi006.odexej (mgr.14176) 852 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:12.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:11 smithi156 bash[29655]: cluster 2023-11-17T05:52:10.803272+0000 mon.smithi006 (mon.0) 879 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:52:13.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:13 smithi156 bash[29655]: cluster 2023-11-17T05:52:12.291264+0000 mgr.smithi006.odexej (mgr.14176) 853 : cluster [DBG] pgmap v603: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T05:52:14.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:13 smithi006 bash[26022]: cluster 2023-11-17T05:52:12.291264+0000 mgr.smithi006.odexej (mgr.14176) 853 : cluster [DBG] pgmap v603: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T05:52:15.071 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:15.072 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:15.693 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:16.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:15 smithi006 bash[26022]: cluster 2023-11-17T05:52:14.292590+0000 mgr.smithi006.odexej (mgr.14176) 854 : cluster [DBG] pgmap v604: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:16.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:15 smithi006 bash[26022]: audit 2023-11-17T05:52:15.051712+0000 mgr.smithi006.odexej (mgr.14176) 855 : audit [DBG] from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:16.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:15 smithi156 bash[29655]: cluster 2023-11-17T05:52:14.292590+0000 mgr.smithi006.odexej (mgr.14176) 854 : cluster [DBG] pgmap v604: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:16.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:15 smithi156 bash[29655]: audit 2023-11-17T05:52:15.051712+0000 mgr.smithi006.odexej (mgr.14176) 855 : audit [DBG] from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:16.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:17.998 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:17 smithi006 bash[26022]: cluster 2023-11-17T05:52:16.293660+0000 mgr.smithi006.odexej (mgr.14176) 856 : cluster [DBG] pgmap v605: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:18.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:17 smithi156 bash[29655]: cluster 2023-11-17T05:52:16.293660+0000 mgr.smithi006.odexej (mgr.14176) 856 : cluster [DBG] pgmap v605: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:19.723 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:19.724 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:19 smithi006 bash[26022]: cluster 2023-11-17T05:52:18.294812+0000 mgr.smithi006.odexej (mgr.14176) 857 : cluster [DBG] pgmap v606: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:19 smithi006 bash[26022]: audit 2023-11-17T05:52:19.719849+0000 mon.smithi006 (mon.0) 880 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:19 smithi156 bash[29655]: cluster 2023-11-17T05:52:18.294812+0000 mgr.smithi006.odexej (mgr.14176) 857 : cluster [DBG] pgmap v606: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:19 smithi156 bash[29655]: audit 2023-11-17T05:52:19.719849+0000 mon.smithi006 (mon.0) 880 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:52:20.348 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:21.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:20 smithi006 bash[26022]: audit 2023-11-17T05:52:19.705817+0000 mgr.smithi006.odexej (mgr.14176) 858 : audit [DBG] from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:21.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:20 smithi156 bash[29655]: audit 2023-11-17T05:52:19.705817+0000 mgr.smithi006.odexej (mgr.14176) 858 : audit [DBG] from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:21.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:22.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:21 smithi006 bash[26022]: cluster 2023-11-17T05:52:20.296220+0000 mgr.smithi006.odexej (mgr.14176) 859 : cluster [DBG] pgmap v607: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:22.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:21 smithi156 bash[29655]: cluster 2023-11-17T05:52:20.296220+0000 mgr.smithi006.odexej (mgr.14176) 859 : cluster [DBG] pgmap v607: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:52:23.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:23 smithi156 bash[29655]: cluster 2023-11-17T05:52:22.297621+0000 mgr.smithi006.odexej (mgr.14176) 860 : cluster [DBG] pgmap v608: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:23 smithi006 bash[26022]: cluster 2023-11-17T05:52:22.297621+0000 mgr.smithi006.odexej (mgr.14176) 860 : cluster [DBG] pgmap v608: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:24.359 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:24.359 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:25.008 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:26.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:26.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:25 smithi006 bash[26022]: cluster 2023-11-17T05:52:24.298979+0000 mgr.smithi006.odexej (mgr.14176) 861 : cluster [DBG] pgmap v609: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:26.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:25 smithi006 bash[26022]: audit 2023-11-17T05:52:24.347654+0000 mgr.smithi006.odexej (mgr.14176) 862 : audit [DBG] from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:26.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:25 smithi156 bash[29655]: cluster 2023-11-17T05:52:24.298979+0000 mgr.smithi006.odexej (mgr.14176) 861 : cluster [DBG] pgmap v609: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:26.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:25 smithi156 bash[29655]: audit 2023-11-17T05:52:24.347654+0000 mgr.smithi006.odexej (mgr.14176) 862 : audit [DBG] from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:28.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:27 smithi006 bash[26022]: cluster 2023-11-17T05:52:26.300081+0000 mgr.smithi006.odexej (mgr.14176) 863 : cluster [DBG] pgmap v610: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:28.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:27 smithi156 bash[29655]: cluster 2023-11-17T05:52:26.300081+0000 mgr.smithi006.odexej (mgr.14176) 863 : cluster [DBG] pgmap v610: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:29.023 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:29.023 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:29.606 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:30.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:29 smithi006 bash[26022]: cluster 2023-11-17T05:52:28.301253+0000 mgr.smithi006.odexej (mgr.14176) 864 : cluster [DBG] pgmap v611: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:30.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:29 smithi006 bash[26022]: audit 2023-11-17T05:52:29.005246+0000 mgr.smithi006.odexej (mgr.14176) 865 : audit [DBG] from='client.14878 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:30.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:29 smithi156 bash[29655]: cluster 2023-11-17T05:52:28.301253+0000 mgr.smithi006.odexej (mgr.14176) 864 : cluster [DBG] pgmap v611: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:30.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:29 smithi156 bash[29655]: audit 2023-11-17T05:52:29.005246+0000 mgr.smithi006.odexej (mgr.14176) 865 : audit [DBG] from='client.14878 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:30.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:31.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:31 smithi006 bash[26022]: cluster 2023-11-17T05:52:30.302623+0000 mgr.smithi006.odexej (mgr.14176) 866 : cluster [DBG] pgmap v612: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:32.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:31 smithi156 bash[29655]: cluster 2023-11-17T05:52:30.302623+0000 mgr.smithi006.odexej (mgr.14176) 866 : cluster [DBG] pgmap v612: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:33.613 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:33.613 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:34.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:33 smithi006 bash[26022]: cluster 2023-11-17T05:52:32.303732+0000 mgr.smithi006.odexej (mgr.14176) 867 : cluster [DBG] pgmap v613: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:34.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:33 smithi156 bash[29655]: cluster 2023-11-17T05:52:32.303732+0000 mgr.smithi006.odexej (mgr.14176) 867 : cluster [DBG] pgmap v613: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:34.227 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:35.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:34 smithi006 bash[26022]: audit 2023-11-17T05:52:33.594065+0000 mgr.smithi006.odexej (mgr.14176) 868 : audit [DBG] from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:35.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:34 smithi156 bash[29655]: audit 2023-11-17T05:52:33.594065+0000 mgr.smithi006.odexej (mgr.14176) 868 : audit [DBG] from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:35.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:35 smithi006 bash[26022]: cluster 2023-11-17T05:52:34.305136+0000 mgr.smithi006.odexej (mgr.14176) 869 : cluster [DBG] pgmap v614: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:36.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:35 smithi156 bash[29655]: cluster 2023-11-17T05:52:34.305136+0000 mgr.smithi006.odexej (mgr.14176) 869 : cluster [DBG] pgmap v614: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:38.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:37 smithi006 bash[26022]: cluster 2023-11-17T05:52:36.305783+0000 mgr.smithi006.odexej (mgr.14176) 870 : cluster [DBG] pgmap v615: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:38.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:37 smithi156 bash[29655]: cluster 2023-11-17T05:52:36.305783+0000 mgr.smithi006.odexej (mgr.14176) 870 : cluster [DBG] pgmap v615: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:38.263 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:38.263 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:38.897 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:39.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:40.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:39 smithi006 bash[26022]: audit 2023-11-17T05:52:38.251801+0000 mgr.smithi006.odexej (mgr.14176) 871 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:40.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:39 smithi006 bash[26022]: cluster 2023-11-17T05:52:38.306860+0000 mgr.smithi006.odexej (mgr.14176) 872 : cluster [DBG] pgmap v616: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:39 smithi156 bash[29655]: audit 2023-11-17T05:52:38.251801+0000 mgr.smithi006.odexej (mgr.14176) 871 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:39 smithi156 bash[29655]: cluster 2023-11-17T05:52:38.306860+0000 mgr.smithi006.odexej (mgr.14176) 872 : cluster [DBG] pgmap v616: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:42.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:41 smithi156 bash[29655]: cluster 2023-11-17T05:52:40.308229+0000 mgr.smithi006.odexej (mgr.14176) 873 : cluster [DBG] pgmap v617: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:42.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:41 smithi006 bash[26022]: cluster 2023-11-17T05:52:40.308229+0000 mgr.smithi006.odexej (mgr.14176) 873 : cluster [DBG] pgmap v617: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:42.939 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:42.940 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:43.498 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:44.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:43 smithi156 bash[29655]: cluster 2023-11-17T05:52:42.309287+0000 mgr.smithi006.odexej (mgr.14176) 874 : cluster [DBG] pgmap v618: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:44.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:43 smithi156 bash[29655]: audit 2023-11-17T05:52:42.919812+0000 mgr.smithi006.odexej (mgr.14176) 875 : audit [DBG] from='client.14890 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:44.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:43 smithi006 bash[26022]: cluster 2023-11-17T05:52:42.309287+0000 mgr.smithi006.odexej (mgr.14176) 874 : cluster [DBG] pgmap v618: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:44.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:43 smithi006 bash[26022]: audit 2023-11-17T05:52:42.919812+0000 mgr.smithi006.odexej (mgr.14176) 875 : audit [DBG] from='client.14890 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:44.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:46.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:45 smithi006 bash[26022]: cluster 2023-11-17T05:52:44.310580+0000 mgr.smithi006.odexej (mgr.14176) 876 : cluster [DBG] pgmap v619: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:46.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:45 smithi156 bash[29655]: cluster 2023-11-17T05:52:44.310580+0000 mgr.smithi006.odexej (mgr.14176) 876 : cluster [DBG] pgmap v619: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:47.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:47.556 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:48.115 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:48.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:47 smithi006 bash[26022]: cluster 2023-11-17T05:52:46.312052+0000 mgr.smithi006.odexej (mgr.14176) 877 : cluster [DBG] pgmap v620: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:47 smithi156 bash[29655]: cluster 2023-11-17T05:52:46.312052+0000 mgr.smithi006.odexej (mgr.14176) 877 : cluster [DBG] pgmap v620: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:49.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:49.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:48 smithi006 bash[26022]: audit 2023-11-17T05:52:47.538959+0000 mgr.smithi006.odexej (mgr.14176) 878 : audit [DBG] from='client.14894 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:49.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:48 smithi156 bash[29655]: audit 2023-11-17T05:52:47.538959+0000 mgr.smithi006.odexej (mgr.14176) 878 : audit [DBG] from='client.14894 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:50.281 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:49 smithi006 bash[26022]: cluster 2023-11-17T05:52:48.313515+0000 mgr.smithi006.odexej (mgr.14176) 879 : cluster [DBG] pgmap v621: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:50.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:49 smithi156 bash[29655]: cluster 2023-11-17T05:52:48.313515+0000 mgr.smithi006.odexej (mgr.14176) 879 : cluster [DBG] pgmap v621: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:52.089 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:52.089 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:52.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:51 smithi006 bash[26022]: cluster 2023-11-17T05:52:50.314890+0000 mgr.smithi006.odexej (mgr.14176) 880 : cluster [DBG] pgmap v622: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:52.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:51 smithi156 bash[29655]: cluster 2023-11-17T05:52:50.314890+0000 mgr.smithi006.odexej (mgr.14176) 880 : cluster [DBG] pgmap v622: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:52.695 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:53.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:52 smithi006 bash[26022]: audit 2023-11-17T05:52:52.077896+0000 mgr.smithi006.odexej (mgr.14176) 881 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:53.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:52 smithi156 bash[29655]: audit 2023-11-17T05:52:52.077896+0000 mgr.smithi006.odexej (mgr.14176) 881 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:53.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:54.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:53 smithi006 bash[26022]: cluster 2023-11-17T05:52:52.315929+0000 mgr.smithi006.odexej (mgr.14176) 882 : cluster [DBG] pgmap v623: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:54.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:53 smithi156 bash[29655]: cluster 2023-11-17T05:52:52.315929+0000 mgr.smithi006.odexej (mgr.14176) 882 : cluster [DBG] pgmap v623: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:52:56.357 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:55 smithi006 bash[26022]: cluster 2023-11-17T05:52:54.316927+0000 mgr.smithi006.odexej (mgr.14176) 883 : cluster [DBG] pgmap v624: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:56.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:55 smithi156 bash[29655]: cluster 2023-11-17T05:52:54.316927+0000 mgr.smithi006.odexej (mgr.14176) 883 : cluster [DBG] pgmap v624: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:56.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:52:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:52:57.411 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:52:58.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:52:58.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:57 smithi006 bash[26022]: cluster 2023-11-17T05:52:56.318009+0000 mgr.smithi006.odexej (mgr.14176) 884 : cluster [DBG] pgmap v625: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:58.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:57 smithi006 bash[26022]: audit 2023-11-17T05:52:56.786887+0000 mgr.smithi006.odexej (mgr.14176) 885 : audit [DBG] from='client.14902 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:52:58.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:57 smithi156 bash[29655]: cluster 2023-11-17T05:52:56.318009+0000 mgr.smithi006.odexej (mgr.14176) 884 : cluster [DBG] pgmap v625: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:52:58.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:57 smithi156 bash[29655]: audit 2023-11-17T05:52:56.786887+0000 mgr.smithi006.odexej (mgr.14176) 885 : audit [DBG] from='client.14902 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:00.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:52:59 smithi006 bash[26022]: cluster 2023-11-17T05:52:58.319185+0000 mgr.smithi006.odexej (mgr.14176) 886 : cluster [DBG] pgmap v626: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:00.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:52:59 smithi156 bash[29655]: cluster 2023-11-17T05:52:58.319185+0000 mgr.smithi006.odexej (mgr.14176) 886 : cluster [DBG] pgmap v626: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:01.496 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:53:01.497 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:53:02.099 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:53:02.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:01 smithi006 bash[26022]: cluster 2023-11-17T05:53:00.320581+0000 mgr.smithi006.odexej (mgr.14176) 887 : cluster [DBG] pgmap v627: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:02.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:01 smithi156 bash[29655]: cluster 2023-11-17T05:53:00.320581+0000 mgr.smithi006.odexej (mgr.14176) 887 : cluster [DBG] pgmap v627: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:03.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:53:03.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:02 smithi006 bash[26022]: audit 2023-11-17T05:53:01.477502+0000 mgr.smithi006.odexej (mgr.14176) 888 : audit [DBG] from='client.14906 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:03.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:03 smithi156 bash[29655]: audit 2023-11-17T05:53:01.477502+0000 mgr.smithi006.odexej (mgr.14176) 888 : audit [DBG] from='client.14906 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:04.293 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:04 smithi006 bash[26022]: cluster 2023-11-17T05:53:02.321637+0000 mgr.smithi006.odexej (mgr.14176) 889 : cluster [DBG] pgmap v628: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:21.931 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:53:21.932 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:50:55.977603Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.rhspla on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:06.524661Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.drplam on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-drplam\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.drplam\nDeploy daemon haproxy.nfs.foo.smithi156.drplam ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.280113Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.pomqkk on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-pomqkk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.pomqkk\nDeploy daemon haproxy.nfs.foo.smithi006.pomqkk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:52:10.284364Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vlvnuy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:53:21.932 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:06 smithi006 bash[26022]: cluster 2023-11-17T05:53:04.322950+0000 mgr.smithi006.odexej (mgr.14176) 890 : cluster [DBG] pgmap v629: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:21.990 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:53:21.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:04 smithi156 bash[29655]: cluster 2023-11-17T05:53:02.321637+0000 mgr.smithi006.odexej (mgr.14176) 889 : cluster [DBG] pgmap v628: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:21.991 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:06 smithi156 bash[29655]: cluster 2023-11-17T05:53:04.322950+0000 mgr.smithi006.odexej (mgr.14176) 890 : cluster [DBG] pgmap v629: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:08 smithi156 bash[29655]: audit 2023-11-17T05:53:06.103370+0000 mgr.smithi006.odexej (mgr.14176) 891 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:08 smithi156 bash[29655]: cluster 2023-11-17T05:53:06.323853+0000 mgr.smithi006.odexej (mgr.14176) 892 : cluster [DBG] pgmap v630: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:10 smithi156 bash[29655]: cluster 2023-11-17T05:53:08.325023+0000 mgr.smithi006.odexej (mgr.14176) 893 : cluster [DBG] pgmap v631: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.292278+0000 mon.smithi006 (mon.0) 881 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.618246+0000 mon.smithi006 (mon.0) 882 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.913547+0000 mon.smithi006 (mon.0) 883 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.915275+0000 mon.smithi006 (mon.0) 884 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.923598+0000 mon.smithi006 (mon.0) 885 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.933300+0000 mon.smithi006 (mon.0) 886 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:11 smithi156 bash[29655]: audit 2023-11-17T05:53:10.948282+0000 mon.smithi006 (mon.0) 887 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:21.992 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:12 smithi156 bash[29655]: cluster 2023-11-17T05:53:10.326216+0000 mgr.smithi006.odexej (mgr.14176) 894 : cluster [DBG] pgmap v632: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:12 smithi156 bash[29655]: cluster 2023-11-17T05:53:10.924467+0000 mgr.smithi006.odexej (mgr.14176) 895 : cluster [DBG] pgmap v633: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:12 smithi156 bash[29655]: cephadm 2023-11-17T05:53:10.952237+0000 mgr.smithi006.odexej (mgr.14176) 896 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dohmqc on smithi156 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:12 smithi156 bash[29655]: cluster 2023-11-17T05:53:11.018697+0000 mon.smithi006 (mon.0) 888 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:12 smithi156 bash[29655]: cluster 2023-11-17T05:53:11.018769+0000 mon.smithi006 (mon.0) 889 : cluster [INF] Cluster is now healthy 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:14 smithi156 bash[29655]: cluster 2023-11-17T05:53:12.925241+0000 mgr.smithi006.odexej (mgr.14176) 897 : cluster [DBG] pgmap v634: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.060 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:15 smithi156 bash[29655]: audit 2023-11-17T05:53:14.691623+0000 mon.smithi006 (mon.0) 890 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dohmqc"}]: dispatch 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: cephadm 2023-11-17T05:53:14.690288+0000 mgr.smithi006.odexej (mgr.14176) 898 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.061 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: cephadm 2023-11-17T05:53:14.690872+0000 mgr.smithi006.odexej (mgr.14176) 899 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: cephadm 2023-11-17T05:53:14.693355+0000 mgr.smithi006.odexej (mgr.14176) 900 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: cephadm 2023-11-17T05:53:14.698503+0000 mgr.smithi006.odexej (mgr.14176) 901 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.sgvscb on smithi006 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:16 smithi156 bash[29655]: cluster 2023-11-17T05:53:14.926159+0000 mgr.smithi006.odexej (mgr.14176) 902 : cluster [DBG] pgmap v635: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:17 smithi156 bash[29655]: cluster 2023-11-17T05:53:16.926918+0000 mgr.smithi006.odexej (mgr.14176) 903 : cluster [DBG] pgmap v636: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:18 smithi156 bash[29655]: audit 2023-11-17T05:53:18.444902+0000 mon.smithi006 (mon.0) 891 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.sgvscb"}]: dispatch 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cephadm 2023-11-17T05:53:18.443798+0000 mgr.smithi006.odexej (mgr.14176) 904 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:53:22.066 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.067 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cephadm 2023-11-17T05:53:18.444245+0000 mgr.smithi006.odexej (mgr.14176) 905 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cephadm 2023-11-17T05:53:18.446577+0000 mgr.smithi006.odexej (mgr.14176) 906 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.135 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.135 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:53:22.135 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.135 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.136 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.136 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.136 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.136 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.136 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.137 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cephadm 2023-11-17T05:53:18.450955+0000 mgr.smithi006.odexej (mgr.14176) 907 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:53:22.137 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cephadm 2023-11-17T05:53:18.452641+0000 mgr.smithi006.odexej (mgr.14176) 908 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:53:22.137 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cluster 2023-11-17T05:53:18.453484+0000 mgr.smithi006.odexej (mgr.14176) 909 : cluster [DBG] pgmap v637: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:53:22.265 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:19 smithi156 bash[29655]: cluster 2023-11-17T05:53:18.521377+0000 mon.smithi006 (mon.0) 892 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:53:22.266 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:21 smithi156 bash[29655]: cluster 2023-11-17T05:53:20.454630+0000 mgr.smithi006.odexej (mgr.14176) 910 : cluster [DBG] pgmap v638: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:53:22.266 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:21 smithi156 bash[29655]: audit 2023-11-17T05:53:20.612896+0000 mon.smithi006 (mon.0) 893 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:22.277 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:08 smithi006 bash[26022]: audit 2023-11-17T05:53:06.103370+0000 mgr.smithi006.odexej (mgr.14176) 891 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:22.277 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:08 smithi006 bash[26022]: cluster 2023-11-17T05:53:06.323853+0000 mgr.smithi006.odexej (mgr.14176) 892 : cluster [DBG] pgmap v630: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:22.324 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:10 smithi006 bash[26022]: cluster 2023-11-17T05:53:08.325023+0000 mgr.smithi006.odexej (mgr.14176) 893 : cluster [DBG] pgmap v631: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:22.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.292278+0000 mon.smithi006 (mon.0) 881 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:53:22.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.618246+0000 mon.smithi006 (mon.0) 882 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:22.330 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.913547+0000 mon.smithi006 (mon.0) 883 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:53:22.330 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.915275+0000 mon.smithi006 (mon.0) 884 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:53:22.330 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.923598+0000 mon.smithi006 (mon.0) 885 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.933300+0000 mon.smithi006 (mon.0) 886 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:11 smithi006 bash[26022]: audit 2023-11-17T05:53:10.948282+0000 mon.smithi006 (mon.0) 887 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:12 smithi006 bash[26022]: cluster 2023-11-17T05:53:10.326216+0000 mgr.smithi006.odexej (mgr.14176) 894 : cluster [DBG] pgmap v632: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:12 smithi006 bash[26022]: cluster 2023-11-17T05:53:10.924467+0000 mgr.smithi006.odexej (mgr.14176) 895 : cluster [DBG] pgmap v633: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:12 smithi006 bash[26022]: cephadm 2023-11-17T05:53:10.952237+0000 mgr.smithi006.odexej (mgr.14176) 896 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dohmqc on smithi156 2023-11-17T05:53:22.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:12 smithi006 bash[26022]: cluster 2023-11-17T05:53:11.018697+0000 mon.smithi006 (mon.0) 888 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:53:22.332 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:12 smithi006 bash[26022]: cluster 2023-11-17T05:53:11.018769+0000 mon.smithi006 (mon.0) 889 : cluster [INF] Cluster is now healthy 2023-11-17T05:53:22.373 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:14 smithi006 bash[26022]: cluster 2023-11-17T05:53:12.925241+0000 mgr.smithi006.odexej (mgr.14176) 897 : cluster [DBG] pgmap v634: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.373 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:15 smithi006 bash[26022]: audit 2023-11-17T05:53:14.691623+0000 mon.smithi006 (mon.0) 890 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dohmqc"}]: dispatch 2023-11-17T05:53:22.373 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: cephadm 2023-11-17T05:53:14.690288+0000 mgr.smithi006.odexej (mgr.14176) 898 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.376 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: cephadm 2023-11-17T05:53:14.690872+0000 mgr.smithi006.odexej (mgr.14176) 899 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: cephadm 2023-11-17T05:53:14.693355+0000 mgr.smithi006.odexej (mgr.14176) 900 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dohmqc ... 2023-11-17T05:53:22.377 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: cephadm 2023-11-17T05:53:14.698503+0000 mgr.smithi006.odexej (mgr.14176) 901 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.sgvscb on smithi006 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:16 smithi006 bash[26022]: cluster 2023-11-17T05:53:14.926159+0000 mgr.smithi006.odexej (mgr.14176) 902 : cluster [DBG] pgmap v635: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:17 smithi006 bash[26022]: cluster 2023-11-17T05:53:16.926918+0000 mgr.smithi006.odexej (mgr.14176) 903 : cluster [DBG] pgmap v636: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:18 smithi006 bash[26022]: audit 2023-11-17T05:53:18.444902+0000 mon.smithi006 (mon.0) 891 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.sgvscb"}]: dispatch 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cephadm 2023-11-17T05:53:18.443798+0000 mgr.smithi006.odexej (mgr.14176) 904 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:53:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cephadm 2023-11-17T05:53:18.444245+0000 mgr.smithi006.odexej (mgr.14176) 905 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cephadm 2023-11-17T05:53:18.446577+0000 mgr.smithi006.odexej (mgr.14176) 906 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb 2023-11-17T05:53:22.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.sgvscb ... 2023-11-17T05:53:22.382 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.428 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:53:22.428 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:53:22.429 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:53:22.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:53:22.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cephadm 2023-11-17T05:53:18.450955+0000 mgr.smithi006.odexej (mgr.14176) 907 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:53:22.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cephadm 2023-11-17T05:53:18.452641+0000 mgr.smithi006.odexej (mgr.14176) 908 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:53:22.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cluster 2023-11-17T05:53:18.453484+0000 mgr.smithi006.odexej (mgr.14176) 909 : cluster [DBG] pgmap v637: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:53:22.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:19 smithi006 bash[26022]: cluster 2023-11-17T05:53:18.521377+0000 mon.smithi006 (mon.0) 892 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:53:22.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:21 smithi006 bash[26022]: cluster 2023-11-17T05:53:20.454630+0000 mgr.smithi006.odexej (mgr.14176) 910 : cluster [DBG] pgmap v638: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T05:53:22.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:21 smithi006 bash[26022]: audit 2023-11-17T05:53:20.612896+0000 mon.smithi006 (mon.0) 893 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:53:22.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:53:23.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:23 smithi006 bash[26022]: cluster 2023-11-17T05:53:22.455658+0000 mgr.smithi006.odexej (mgr.14176) 911 : cluster [DBG] pgmap v639: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:53:54.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:25 smithi006 bash[26022]: cluster 2023-11-17T05:53:24.456620+0000 mgr.smithi006.odexej (mgr.14176) 912 : cluster [DBG] pgmap v640: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.707 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:53:54.707 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:53:54.708 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:53:54.709 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:23 smithi156 bash[29655]: cluster 2023-11-17T05:53:22.455658+0000 mgr.smithi006.odexej (mgr.14176) 911 : cluster [DBG] pgmap v639: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:25 smithi156 bash[29655]: cluster 2023-11-17T05:53:24.456620+0000 mgr.smithi006.odexej (mgr.14176) 912 : cluster [DBG] pgmap v640: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:27 smithi156 bash[29655]: audit 2023-11-17T05:53:26.138566+0000 mgr.smithi006.odexej (mgr.14176) 913 : audit [DBG] from='client.14914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:27 smithi156 bash[29655]: cluster 2023-11-17T05:53:26.457618+0000 mgr.smithi006.odexej (mgr.14176) 914 : cluster [DBG] pgmap v641: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:29 smithi156 bash[29655]: cluster 2023-11-17T05:53:28.458798+0000 mgr.smithi006.odexej (mgr.14176) 915 : cluster [DBG] pgmap v642: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:31 smithi156 bash[29655]: cluster 2023-11-17T05:53:30.460226+0000 mgr.smithi006.odexej (mgr.14176) 916 : cluster [DBG] pgmap v643: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:33 smithi156 bash[29655]: cluster 2023-11-17T05:53:32.461432+0000 mgr.smithi006.odexej (mgr.14176) 917 : cluster [DBG] pgmap v644: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:54.710 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:35 smithi156 bash[29655]: cluster 2023-11-17T05:53:34.462862+0000 mgr.smithi006.odexej (mgr.14176) 918 : cluster [DBG] pgmap v645: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:37 smithi156 bash[29655]: cluster 2023-11-17T05:53:36.463907+0000 mgr.smithi006.odexej (mgr.14176) 919 : cluster [DBG] pgmap v646: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:39 smithi156 bash[29655]: cluster 2023-11-17T05:53:38.465127+0000 mgr.smithi006.odexej (mgr.14176) 920 : cluster [DBG] pgmap v647: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:41 smithi156 bash[29655]: cluster 2023-11-17T05:53:40.466511+0000 mgr.smithi006.odexej (mgr.14176) 921 : cluster [DBG] pgmap v648: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:43 smithi156 bash[29655]: cluster 2023-11-17T05:53:42.467630+0000 mgr.smithi006.odexej (mgr.14176) 922 : cluster [DBG] pgmap v649: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:45 smithi156 bash[29655]: cluster 2023-11-17T05:53:44.469009+0000 mgr.smithi006.odexej (mgr.14176) 923 : cluster [DBG] pgmap v650: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:47 smithi156 bash[29655]: cluster 2023-11-17T05:53:46.470054+0000 mgr.smithi006.odexej (mgr.14176) 924 : cluster [DBG] pgmap v651: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:49 smithi156 bash[29655]: cluster 2023-11-17T05:53:48.471233+0000 mgr.smithi006.odexej (mgr.14176) 925 : cluster [DBG] pgmap v652: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:51 smithi156 bash[29655]: cluster 2023-11-17T05:53:50.472623+0000 mgr.smithi006.odexej (mgr.14176) 926 : cluster [DBG] pgmap v653: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.742 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:53 smithi156 bash[29655]: cluster 2023-11-17T05:53:52.473614+0000 mgr.smithi006.odexej (mgr.14176) 927 : cluster [DBG] pgmap v654: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:54.743 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:27 smithi006 bash[26022]: audit 2023-11-17T05:53:26.138566+0000 mgr.smithi006.odexej (mgr.14176) 913 : audit [DBG] from='client.14914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:53:54.743 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:27 smithi006 bash[26022]: cluster 2023-11-17T05:53:26.457618+0000 mgr.smithi006.odexej (mgr.14176) 914 : cluster [DBG] pgmap v641: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.743 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:29 smithi006 bash[26022]: cluster 2023-11-17T05:53:28.458798+0000 mgr.smithi006.odexej (mgr.14176) 915 : cluster [DBG] pgmap v642: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:53:54.743 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:31 smithi006 bash[26022]: cluster 2023-11-17T05:53:30.460226+0000 mgr.smithi006.odexej (mgr.14176) 916 : cluster [DBG] pgmap v643: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:33 smithi006 bash[26022]: cluster 2023-11-17T05:53:32.461432+0000 mgr.smithi006.odexej (mgr.14176) 917 : cluster [DBG] pgmap v644: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:35 smithi006 bash[26022]: cluster 2023-11-17T05:53:34.462862+0000 mgr.smithi006.odexej (mgr.14176) 918 : cluster [DBG] pgmap v645: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:37 smithi006 bash[26022]: cluster 2023-11-17T05:53:36.463907+0000 mgr.smithi006.odexej (mgr.14176) 919 : cluster [DBG] pgmap v646: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:39 smithi006 bash[26022]: cluster 2023-11-17T05:53:38.465127+0000 mgr.smithi006.odexej (mgr.14176) 920 : cluster [DBG] pgmap v647: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:41 smithi006 bash[26022]: cluster 2023-11-17T05:53:40.466511+0000 mgr.smithi006.odexej (mgr.14176) 921 : cluster [DBG] pgmap v648: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:43 smithi006 bash[26022]: cluster 2023-11-17T05:53:42.467630+0000 mgr.smithi006.odexej (mgr.14176) 922 : cluster [DBG] pgmap v649: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:45 smithi006 bash[26022]: cluster 2023-11-17T05:53:44.469009+0000 mgr.smithi006.odexej (mgr.14176) 923 : cluster [DBG] pgmap v650: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:47 smithi006 bash[26022]: cluster 2023-11-17T05:53:46.470054+0000 mgr.smithi006.odexej (mgr.14176) 924 : cluster [DBG] pgmap v651: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:49 smithi006 bash[26022]: cluster 2023-11-17T05:53:48.471233+0000 mgr.smithi006.odexej (mgr.14176) 925 : cluster [DBG] pgmap v652: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:51 smithi006 bash[26022]: cluster 2023-11-17T05:53:50.472623+0000 mgr.smithi006.odexej (mgr.14176) 926 : cluster [DBG] pgmap v653: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:54.744 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:53 smithi006 bash[26022]: cluster 2023-11-17T05:53:52.473614+0000 mgr.smithi006.odexej (mgr.14176) 927 : cluster [DBG] pgmap v654: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:53:55.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:53:56.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:55 smithi006 bash[26022]: cluster 2023-11-17T05:53:54.474989+0000 mgr.smithi006.odexej (mgr.14176) 928 : cluster [DBG] pgmap v655: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:56.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:55 smithi156 bash[29655]: cluster 2023-11-17T05:53:54.474989+0000 mgr.smithi006.odexej (mgr.14176) 928 : cluster [DBG] pgmap v655: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:58.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:57 smithi006 bash[26022]: cluster 2023-11-17T05:53:56.476071+0000 mgr.smithi006.odexej (mgr.14176) 929 : cluster [DBG] pgmap v656: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:58.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:57 smithi156 bash[29655]: cluster 2023-11-17T05:53:56.476071+0000 mgr.smithi006.odexej (mgr.14176) 929 : cluster [DBG] pgmap v656: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:58.972 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:53:58.973 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:53:59.628 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:53:59.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:59 smithi006 bash[26022]: cluster 2023-11-17T05:53:58.476989+0000 mgr.smithi006.odexej (mgr.14176) 930 : cluster [DBG] pgmap v657: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:53:59.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:53:59 smithi006 bash[26022]: audit 2023-11-17T05:53:58.953656+0000 mgr.smithi006.odexej (mgr.14176) 931 : audit [DBG] from='client.14918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:00.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:59 smithi156 bash[29655]: cluster 2023-11-17T05:53:58.476989+0000 mgr.smithi006.odexej (mgr.14176) 930 : cluster [DBG] pgmap v657: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:00.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:53:59 smithi156 bash[29655]: audit 2023-11-17T05:53:58.953656+0000 mgr.smithi006.odexej (mgr.14176) 931 : audit [DBG] from='client.14918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:00.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:01.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:01 smithi006 bash[26022]: cluster 2023-11-17T05:54:00.478384+0000 mgr.smithi006.odexej (mgr.14176) 932 : cluster [DBG] pgmap v658: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:02.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:01 smithi156 bash[29655]: cluster 2023-11-17T05:54:00.478384+0000 mgr.smithi006.odexej (mgr.14176) 932 : cluster [DBG] pgmap v658: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:03.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:03.787 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:03.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:03 smithi156 bash[29655]: cluster 2023-11-17T05:54:02.479355+0000 mgr.smithi006.odexej (mgr.14176) 933 : cluster [DBG] pgmap v659: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:04.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:03 smithi006 bash[26022]: cluster 2023-11-17T05:54:02.479355+0000 mgr.smithi006.odexej (mgr.14176) 933 : cluster [DBG] pgmap v659: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:04.409 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:05.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:04 smithi006 bash[26022]: audit 2023-11-17T05:54:03.768359+0000 mgr.smithi006.odexej (mgr.14176) 934 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:05.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:04 smithi156 bash[29655]: audit 2023-11-17T05:54:03.768359+0000 mgr.smithi006.odexej (mgr.14176) 934 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:05.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:06.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:05 smithi006 bash[26022]: cluster 2023-11-17T05:54:04.480811+0000 mgr.smithi006.odexej (mgr.14176) 935 : cluster [DBG] pgmap v660: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:06.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:05 smithi156 bash[29655]: cluster 2023-11-17T05:54:04.480811+0000 mgr.smithi006.odexej (mgr.14176) 935 : cluster [DBG] pgmap v660: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:08.100 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:07 smithi006 bash[26022]: cluster 2023-11-17T05:54:06.481927+0000 mgr.smithi006.odexej (mgr.14176) 936 : cluster [DBG] pgmap v661: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:08.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:07 smithi156 bash[29655]: cluster 2023-11-17T05:54:06.481927+0000 mgr.smithi006.odexej (mgr.14176) 936 : cluster [DBG] pgmap v661: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:08.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:08.565 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:09.190 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:10.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:09 smithi006 bash[26022]: cluster 2023-11-17T05:54:08.483149+0000 mgr.smithi006.odexej (mgr.14176) 937 : cluster [DBG] pgmap v662: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:10.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:09 smithi006 bash[26022]: audit 2023-11-17T05:54:08.546656+0000 mgr.smithi006.odexej (mgr.14176) 938 : audit [DBG] from='client.14926 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:10.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:09 smithi156 bash[29655]: cluster 2023-11-17T05:54:08.483149+0000 mgr.smithi006.odexej (mgr.14176) 937 : cluster [DBG] pgmap v662: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:10.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:09 smithi156 bash[29655]: audit 2023-11-17T05:54:08.546656+0000 mgr.smithi006.odexej (mgr.14176) 938 : audit [DBG] from='client.14926 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:10.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:12.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:11 smithi006 bash[26022]: cluster 2023-11-17T05:54:10.484584+0000 mgr.smithi006.odexej (mgr.14176) 939 : cluster [DBG] pgmap v663: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:11 smithi156 bash[29655]: cluster 2023-11-17T05:54:10.484584+0000 mgr.smithi006.odexej (mgr.14176) 939 : cluster [DBG] pgmap v663: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:13.270 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:13.270 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:13.833 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:13.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:13 smithi156 bash[29655]: cluster 2023-11-17T05:54:12.485619+0000 mgr.smithi006.odexej (mgr.14176) 940 : cluster [DBG] pgmap v664: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:14.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:13 smithi006 bash[26022]: cluster 2023-11-17T05:54:12.485619+0000 mgr.smithi006.odexej (mgr.14176) 940 : cluster [DBG] pgmap v664: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:14.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:15.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:14 smithi006 bash[26022]: audit 2023-11-17T05:54:13.251560+0000 mgr.smithi006.odexej (mgr.14176) 941 : audit [DBG] from='client.14930 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:15.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:14 smithi156 bash[29655]: audit 2023-11-17T05:54:13.251560+0000 mgr.smithi006.odexej (mgr.14176) 941 : audit [DBG] from='client.14930 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:16.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:15 smithi006 bash[26022]: cluster 2023-11-17T05:54:14.486989+0000 mgr.smithi006.odexej (mgr.14176) 942 : cluster [DBG] pgmap v665: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:16.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:15 smithi156 bash[29655]: cluster 2023-11-17T05:54:14.486989+0000 mgr.smithi006.odexej (mgr.14176) 942 : cluster [DBG] pgmap v665: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:17.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:17.892 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:18.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:17 smithi006 bash[26022]: cluster 2023-11-17T05:54:16.488095+0000 mgr.smithi006.odexej (mgr.14176) 943 : cluster [DBG] pgmap v666: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:18.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:17 smithi156 bash[29655]: cluster 2023-11-17T05:54:16.488095+0000 mgr.smithi006.odexej (mgr.14176) 943 : cluster [DBG] pgmap v666: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:18.505 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:19.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:18 smithi006 bash[26022]: audit 2023-11-17T05:54:17.873482+0000 mgr.smithi006.odexej (mgr.14176) 944 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:19.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:18 smithi006 bash[26022]: audit 2023-11-17T05:54:18.456551+0000 mon.smithi006 (mon.0) 894 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:54:19.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:18 smithi006 bash[26022]: audit 2023-11-17T05:54:18.779609+0000 mon.smithi006 (mon.0) 895 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:54:19.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:18 smithi006 bash[26022]: audit 2023-11-17T05:54:18.781168+0000 mon.smithi006 (mon.0) 896 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:54:19.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:18 smithi006 bash[26022]: audit 2023-11-17T05:54:18.790804+0000 mon.smithi006 (mon.0) 897 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:18 smithi156 bash[29655]: audit 2023-11-17T05:54:17.873482+0000 mgr.smithi006.odexej (mgr.14176) 944 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:18 smithi156 bash[29655]: audit 2023-11-17T05:54:18.456551+0000 mon.smithi006 (mon.0) 894 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:54:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:18 smithi156 bash[29655]: audit 2023-11-17T05:54:18.779609+0000 mon.smithi006 (mon.0) 895 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:54:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:18 smithi156 bash[29655]: audit 2023-11-17T05:54:18.781168+0000 mon.smithi006 (mon.0) 896 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:54:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:18 smithi156 bash[29655]: audit 2023-11-17T05:54:18.790804+0000 mon.smithi006 (mon.0) 897 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:19.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: cluster 2023-11-17T05:54:18.488984+0000 mgr.smithi006.odexej (mgr.14176) 945 : cluster [DBG] pgmap v667: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: cluster 2023-11-17T05:54:18.791741+0000 mgr.smithi006.odexej (mgr.14176) 946 : cluster [DBG] pgmap v668: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: cluster 2023-11-17T05:54:18.797800+0000 mon.smithi006 (mon.0) 898 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: cluster 2023-11-17T05:54:18.797856+0000 mon.smithi006 (mon.0) 899 : cluster [INF] Cluster is now healthy 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: audit 2023-11-17T05:54:18.803382+0000 mon.smithi006 (mon.0) 900 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: audit 2023-11-17T05:54:18.819182+0000 mon.smithi006 (mon.0) 901 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:20.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: cephadm 2023-11-17T05:54:18.823435+0000 mgr.smithi006.odexej (mgr.14176) 947 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xanutw on smithi156 2023-11-17T05:54:20.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:19 smithi006 bash[26022]: audit 2023-11-17T05:54:19.720454+0000 mon.smithi006 (mon.0) 902 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: cluster 2023-11-17T05:54:18.488984+0000 mgr.smithi006.odexej (mgr.14176) 945 : cluster [DBG] pgmap v667: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: cluster 2023-11-17T05:54:18.791741+0000 mgr.smithi006.odexej (mgr.14176) 946 : cluster [DBG] pgmap v668: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: cluster 2023-11-17T05:54:18.797800+0000 mon.smithi006 (mon.0) 898 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:54:20.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: cluster 2023-11-17T05:54:18.797856+0000 mon.smithi006 (mon.0) 899 : cluster [INF] Cluster is now healthy 2023-11-17T05:54:20.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: audit 2023-11-17T05:54:18.803382+0000 mon.smithi006 (mon.0) 900 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:54:20.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: audit 2023-11-17T05:54:18.819182+0000 mon.smithi006 (mon.0) 901 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:20.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: cephadm 2023-11-17T05:54:18.823435+0000 mgr.smithi006.odexej (mgr.14176) 947 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xanutw on smithi156 2023-11-17T05:54:20.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:19 smithi156 bash[29655]: audit 2023-11-17T05:54:19.720454+0000 mon.smithi006 (mon.0) 902 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:22.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:21 smithi006 bash[26022]: cluster 2023-11-17T05:54:20.792790+0000 mgr.smithi006.odexej (mgr.14176) 948 : cluster [DBG] pgmap v669: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:22.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:21 smithi156 bash[29655]: cluster 2023-11-17T05:54:20.792790+0000 mgr.smithi006.odexej (mgr.14176) 948 : cluster [DBG] pgmap v669: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:22.598 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:22.599 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:52:10.288315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.mlvcer on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:14.693064Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dohmqc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dohmqc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dohmqc\nDeploy daemon haproxy.nfs.foo.smithi156.dohmqc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.446319Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.sgvscb on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-sgvscb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.sgvscb\nDeploy daemon haproxy.nfs.foo.smithi006.sgvscb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:53:18.450839Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.vgtrxm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:22.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:22 smithi006 bash[26022]: audit 2023-11-17T05:54:22.609982+0000 mon.smithi006 (mon.0) 903 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xanutw"}]: dispatch 2023-11-17T05:54:23.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:22 smithi156 bash[29655]: audit 2023-11-17T05:54:22.609982+0000 mon.smithi006 (mon.0) 903 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xanutw"}]: dispatch 2023-11-17T05:54:23.246 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: audit 2023-11-17T05:54:22.582762+0000 mgr.smithi006.odexej (mgr.14176) 949 : audit [DBG] from='client.14938 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: cephadm 2023-11-17T05:54:22.609199+0000 mgr.smithi006.odexej (mgr.14176) 950 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:54:23.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: cephadm 2023-11-17T05:54:22.609503+0000 mgr.smithi006.odexej (mgr.14176) 951 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: cephadm 2023-11-17T05:54:22.611034+0000 mgr.smithi006.odexej (mgr.14176) 952 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: cephadm 2023-11-17T05:54:22.614200+0000 mgr.smithi006.odexej (mgr.14176) 953 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.qgzvtd on smithi006 2023-11-17T05:54:23.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:23 smithi156 bash[29655]: cluster 2023-11-17T05:54:22.793557+0000 mgr.smithi006.odexej (mgr.14176) 954 : cluster [DBG] pgmap v670: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: audit 2023-11-17T05:54:22.582762+0000 mgr.smithi006.odexej (mgr.14176) 949 : audit [DBG] from='client.14938 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: cephadm 2023-11-17T05:54:22.609199+0000 mgr.smithi006.odexej (mgr.14176) 950 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:24.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:24.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:24.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:54:24.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:24.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:24.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:24.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:24.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.237 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: cephadm 2023-11-17T05:54:22.609503+0000 mgr.smithi006.odexej (mgr.14176) 951 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: cephadm 2023-11-17T05:54:22.611034+0000 mgr.smithi006.odexej (mgr.14176) 952 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xanutw ... 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:24.238 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:24.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:24.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: cephadm 2023-11-17T05:54:22.614200+0000 mgr.smithi006.odexej (mgr.14176) 953 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.qgzvtd on smithi006 2023-11-17T05:54:24.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:23 smithi006 bash[26022]: cluster 2023-11-17T05:54:22.793557+0000 mgr.smithi006.odexej (mgr.14176) 954 : cluster [DBG] pgmap v670: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:24.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:26.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:25 smithi006 bash[26022]: cluster 2023-11-17T05:54:24.794860+0000 mgr.smithi006.odexej (mgr.14176) 955 : cluster [DBG] pgmap v671: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:30.099 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:30.103 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cephadm 2023-11-17T05:54:26.763820+0000 mgr.smithi006.odexej (mgr.14176) 956 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.134 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cephadm 2023-11-17T05:54:26.764322+0000 mgr.smithi006.odexej (mgr.14176) 957 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: audit 2023-11-17T05:54:26.764890+0000 mon.smithi006 (mon.0) 904 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.qgzvtd"}]: dispatch 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cephadm 2023-11-17T05:54:26.766370+0000 mgr.smithi006.odexej (mgr.14176) 958 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.178 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.179 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.179 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cephadm 2023-11-17T05:54:26.770145+0000 mgr.smithi006.odexej (mgr.14176) 959 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:54:30.179 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cephadm 2023-11-17T05:54:26.772222+0000 mgr.smithi006.odexej (mgr.14176) 960 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:54:30.179 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:27 smithi006 bash[26022]: cluster 2023-11-17T05:54:26.773607+0000 mgr.smithi006.odexej (mgr.14176) 961 : cluster [DBG] pgmap v672: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:30.179 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:25 smithi156 bash[29655]: cluster 2023-11-17T05:54:24.794860+0000 mgr.smithi006.odexej (mgr.14176) 955 : cluster [DBG] pgmap v671: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cephadm 2023-11-17T05:54:26.763820+0000 mgr.smithi006.odexej (mgr.14176) 956 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cephadm 2023-11-17T05:54:26.764322+0000 mgr.smithi006.odexej (mgr.14176) 957 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: audit 2023-11-17T05:54:26.764890+0000 mon.smithi006 (mon.0) 904 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.qgzvtd"}]: dispatch 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cephadm 2023-11-17T05:54:26.766370+0000 mgr.smithi006.odexej (mgr.14176) 958 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd 2023-11-17T05:54:30.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.qgzvtd ... 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cephadm 2023-11-17T05:54:26.770145+0000 mgr.smithi006.odexej (mgr.14176) 959 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cephadm 2023-11-17T05:54:26.772222+0000 mgr.smithi006.odexej (mgr.14176) 960 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:27 smithi156 bash[29655]: cluster 2023-11-17T05:54:26.773607+0000 mgr.smithi006.odexej (mgr.14176) 961 : cluster [DBG] pgmap v672: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:30.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:28 smithi156 bash[29655]: audit 2023-11-17T05:54:27.393854+0000 mgr.smithi006.odexej (mgr.14176) 962 : audit [DBG] from='client.14942 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:30.273 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:28 smithi156 bash[29655]: cluster 2023-11-17T05:54:27.535049+0000 mon.smithi006 (mon.0) 905 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:54:30.273 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:29 smithi156 bash[29655]: cluster 2023-11-17T05:54:28.775129+0000 mgr.smithi006.odexej (mgr.14176) 963 : cluster [DBG] pgmap v673: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:30.274 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:30.274 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:28 smithi006 bash[26022]: audit 2023-11-17T05:54:27.393854+0000 mgr.smithi006.odexej (mgr.14176) 962 : audit [DBG] from='client.14942 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:30.275 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:28 smithi006 bash[26022]: cluster 2023-11-17T05:54:27.535049+0000 mon.smithi006 (mon.0) 905 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:54:30.275 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:29 smithi006 bash[26022]: cluster 2023-11-17T05:54:28.775129+0000 mgr.smithi006.odexej (mgr.14176) 963 : cluster [DBG] pgmap v673: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:54:31.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:31.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:31 smithi156 bash[29655]: audit 2023-11-17T05:54:30.668549+0000 mon.smithi006 (mon.0) 906 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:31.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:31 smithi156 bash[29655]: cluster 2023-11-17T05:54:30.776645+0000 mgr.smithi006.odexej (mgr.14176) 964 : cluster [DBG] pgmap v674: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:32.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:31 smithi006 bash[26022]: audit 2023-11-17T05:54:30.668549+0000 mon.smithi006 (mon.0) 906 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:32.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:31 smithi006 bash[26022]: cluster 2023-11-17T05:54:30.776645+0000 mgr.smithi006.odexej (mgr.14176) 964 : cluster [DBG] pgmap v674: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:33.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:33 smithi156 bash[29655]: cluster 2023-11-17T05:54:32.777745+0000 mgr.smithi006.odexej (mgr.14176) 965 : cluster [DBG] pgmap v675: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:34.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:33 smithi006 bash[26022]: cluster 2023-11-17T05:54:32.777745+0000 mgr.smithi006.odexej (mgr.14176) 965 : cluster [DBG] pgmap v675: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:34.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:34.361 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:34.972 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:35.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:35.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:35 smithi006 bash[26022]: audit 2023-11-17T05:54:34.343757+0000 mgr.smithi006.odexej (mgr.14176) 966 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:35.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:35 smithi006 bash[26022]: audit 2023-11-17T05:54:34.724344+0000 mon.smithi006 (mon.0) 907 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:35.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:35 smithi006 bash[26022]: cluster 2023-11-17T05:54:34.779137+0000 mgr.smithi006.odexej (mgr.14176) 967 : cluster [DBG] pgmap v676: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:36.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:35 smithi156 bash[29655]: audit 2023-11-17T05:54:34.343757+0000 mgr.smithi006.odexej (mgr.14176) 966 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:36.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:35 smithi156 bash[29655]: audit 2023-11-17T05:54:34.724344+0000 mon.smithi006 (mon.0) 907 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:54:36.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:35 smithi156 bash[29655]: cluster 2023-11-17T05:54:34.779137+0000 mgr.smithi006.odexej (mgr.14176) 967 : cluster [DBG] pgmap v676: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:37.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:37 smithi006 bash[26022]: cluster 2023-11-17T05:54:36.780229+0000 mgr.smithi006.odexej (mgr.14176) 968 : cluster [DBG] pgmap v677: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:37.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:37 smithi156 bash[29655]: cluster 2023-11-17T05:54:36.780229+0000 mgr.smithi006.odexej (mgr.14176) 968 : cluster [DBG] pgmap v677: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:39.134 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:39.135 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:39.685 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:40.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:39 smithi006 bash[26022]: cluster 2023-11-17T05:54:38.781509+0000 mgr.smithi006.odexej (mgr.14176) 969 : cluster [DBG] pgmap v678: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:40.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:39 smithi156 bash[29655]: cluster 2023-11-17T05:54:38.781509+0000 mgr.smithi006.odexej (mgr.14176) 969 : cluster [DBG] pgmap v678: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:40.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:41.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:40 smithi006 bash[26022]: audit 2023-11-17T05:54:39.118360+0000 mgr.smithi006.odexej (mgr.14176) 970 : audit [DBG] from='client.14950 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:41.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:40 smithi156 bash[29655]: audit 2023-11-17T05:54:39.118360+0000 mgr.smithi006.odexej (mgr.14176) 970 : audit [DBG] from='client.14950 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:42.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:41 smithi006 bash[26022]: cluster 2023-11-17T05:54:40.782880+0000 mgr.smithi006.odexej (mgr.14176) 971 : cluster [DBG] pgmap v679: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:42.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:41 smithi156 bash[29655]: cluster 2023-11-17T05:54:40.782880+0000 mgr.smithi006.odexej (mgr.14176) 971 : cluster [DBG] pgmap v679: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:43.745 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:43.745 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:43.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:43 smithi156 bash[29655]: cluster 2023-11-17T05:54:42.783962+0000 mgr.smithi006.odexej (mgr.14176) 972 : cluster [DBG] pgmap v680: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:44.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:43 smithi006 bash[26022]: cluster 2023-11-17T05:54:42.783962+0000 mgr.smithi006.odexej (mgr.14176) 972 : cluster [DBG] pgmap v680: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:54:44.381 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:45.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:44 smithi006 bash[26022]: audit 2023-11-17T05:54:43.730243+0000 mgr.smithi006.odexej (mgr.14176) 973 : audit [DBG] from='client.14954 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:45.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:44 smithi156 bash[29655]: audit 2023-11-17T05:54:43.730243+0000 mgr.smithi006.odexej (mgr.14176) 973 : audit [DBG] from='client.14954 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:54:45.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:54:46.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:45 smithi006 bash[26022]: cluster 2023-11-17T05:54:44.785519+0000 mgr.smithi006.odexej (mgr.14176) 974 : cluster [DBG] pgmap v681: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:46.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:45 smithi156 bash[29655]: cluster 2023-11-17T05:54:44.785519+0000 mgr.smithi006.odexej (mgr.14176) 974 : cluster [DBG] pgmap v681: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:47.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:47 smithi006 bash[26022]: cluster 2023-11-17T05:54:46.786581+0000 mgr.smithi006.odexej (mgr.14176) 975 : cluster [DBG] pgmap v682: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:47.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:47 smithi156 bash[29655]: cluster 2023-11-17T05:54:46.786581+0000 mgr.smithi006.odexej (mgr.14176) 975 : cluster [DBG] pgmap v682: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:54:48.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:54:48.387 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:54:48.991 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:54:49.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:01.172 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:49 smithi156 bash[29655]: audit 2023-11-17T05:54:48.368592+0000 mgr.smithi006.odexej (mgr.14176) 976 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:01.172 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:49 smithi156 bash[29655]: cluster 2023-11-17T05:54:48.787762+0000 mgr.smithi006.odexej (mgr.14176) 977 : cluster [DBG] pgmap v683: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.172 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:51 smithi156 bash[29655]: cluster 2023-11-17T05:54:50.789137+0000 mgr.smithi006.odexej (mgr.14176) 978 : cluster [DBG] pgmap v684: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.172 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:53 smithi156 bash[29655]: cluster 2023-11-17T05:54:52.790164+0000 mgr.smithi006.odexej (mgr.14176) 979 : cluster [DBG] pgmap v685: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:01.172 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:55 smithi156 bash[29655]: cluster 2023-11-17T05:54:54.791603+0000 mgr.smithi006.odexej (mgr.14176) 980 : cluster [DBG] pgmap v686: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.254 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:57 smithi156 bash[29655]: cluster 2023-11-17T05:54:56.792624+0000 mgr.smithi006.odexej (mgr.14176) 981 : cluster [DBG] pgmap v687: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.254 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:54:59 smithi156 bash[29655]: cluster 2023-11-17T05:54:58.793908+0000 mgr.smithi006.odexej (mgr.14176) 982 : cluster [DBG] pgmap v688: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.254 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:49 smithi006 bash[26022]: audit 2023-11-17T05:54:48.368592+0000 mgr.smithi006.odexej (mgr.14176) 976 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:01.254 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:49 smithi006 bash[26022]: cluster 2023-11-17T05:54:48.787762+0000 mgr.smithi006.odexej (mgr.14176) 977 : cluster [DBG] pgmap v683: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:51 smithi006 bash[26022]: cluster 2023-11-17T05:54:50.789137+0000 mgr.smithi006.odexej (mgr.14176) 978 : cluster [DBG] pgmap v684: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:53 smithi006 bash[26022]: cluster 2023-11-17T05:54:52.790164+0000 mgr.smithi006.odexej (mgr.14176) 979 : cluster [DBG] pgmap v685: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:01.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:55 smithi006 bash[26022]: cluster 2023-11-17T05:54:54.791603+0000 mgr.smithi006.odexej (mgr.14176) 980 : cluster [DBG] pgmap v686: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:57 smithi006 bash[26022]: cluster 2023-11-17T05:54:56.792624+0000 mgr.smithi006.odexej (mgr.14176) 981 : cluster [DBG] pgmap v687: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:01.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:54:59 smithi006 bash[26022]: cluster 2023-11-17T05:54:58.793908+0000 mgr.smithi006.odexej (mgr.14176) 982 : cluster [DBG] pgmap v688: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:02.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:01 smithi006 bash[26022]: cluster 2023-11-17T05:55:00.795354+0000 mgr.smithi006.odexej (mgr.14176) 983 : cluster [DBG] pgmap v689: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:02.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:01 smithi156 bash[29655]: cluster 2023-11-17T05:55:00.795354+0000 mgr.smithi006.odexej (mgr.14176) 983 : cluster [DBG] pgmap v689: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:03.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:03 smithi156 bash[29655]: cluster 2023-11-17T05:55:02.796552+0000 mgr.smithi006.odexej (mgr.14176) 984 : cluster [DBG] pgmap v690: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:04.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:03 smithi006 bash[26022]: cluster 2023-11-17T05:55:02.796552+0000 mgr.smithi006.odexej (mgr.14176) 984 : cluster [DBG] pgmap v690: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:04.559 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:05.131 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:06.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:06.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:05 smithi006 bash[26022]: audit 2023-11-17T05:55:04.545513+0000 mgr.smithi006.odexej (mgr.14176) 985 : audit [DBG] from='client.14962 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:06.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:05 smithi006 bash[26022]: cluster 2023-11-17T05:55:04.797960+0000 mgr.smithi006.odexej (mgr.14176) 986 : cluster [DBG] pgmap v691: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:06.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:05 smithi156 bash[29655]: audit 2023-11-17T05:55:04.545513+0000 mgr.smithi006.odexej (mgr.14176) 985 : audit [DBG] from='client.14962 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:06.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:05 smithi156 bash[29655]: cluster 2023-11-17T05:55:04.797960+0000 mgr.smithi006.odexej (mgr.14176) 986 : cluster [DBG] pgmap v691: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:07.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:07 smithi006 bash[26022]: cluster 2023-11-17T05:55:06.799074+0000 mgr.smithi006.odexej (mgr.14176) 987 : cluster [DBG] pgmap v692: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:07.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:07 smithi156 bash[29655]: cluster 2023-11-17T05:55:06.799074+0000 mgr.smithi006.odexej (mgr.14176) 987 : cluster [DBG] pgmap v692: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:09.128 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:09.128 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:09.690 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:10.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:09 smithi006 bash[26022]: cluster 2023-11-17T05:55:08.800537+0000 mgr.smithi006.odexej (mgr.14176) 988 : cluster [DBG] pgmap v693: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:10.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:09 smithi156 bash[29655]: cluster 2023-11-17T05:55:08.800537+0000 mgr.smithi006.odexej (mgr.14176) 988 : cluster [DBG] pgmap v693: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:10.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:11.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:10 smithi006 bash[26022]: audit 2023-11-17T05:55:09.117367+0000 mgr.smithi006.odexej (mgr.14176) 989 : audit [DBG] from='client.14966 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:11.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:10 smithi156 bash[29655]: audit 2023-11-17T05:55:09.117367+0000 mgr.smithi006.odexej (mgr.14176) 989 : audit [DBG] from='client.14966 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:11.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:11 smithi006 bash[26022]: cluster 2023-11-17T05:55:10.802039+0000 mgr.smithi006.odexej (mgr.14176) 990 : cluster [DBG] pgmap v694: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:12.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:11 smithi156 bash[29655]: cluster 2023-11-17T05:55:10.802039+0000 mgr.smithi006.odexej (mgr.14176) 990 : cluster [DBG] pgmap v694: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:13.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:13.651 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:14.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:13 smithi006 bash[26022]: cluster 2023-11-17T05:55:12.803118+0000 mgr.smithi006.odexej (mgr.14176) 991 : cluster [DBG] pgmap v695: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:14.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:13 smithi156 bash[29655]: cluster 2023-11-17T05:55:12.803118+0000 mgr.smithi006.odexej (mgr.14176) 991 : cluster [DBG] pgmap v695: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:14.215 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:15.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:14 smithi006 bash[26022]: audit 2023-11-17T05:55:13.636010+0000 mgr.smithi006.odexej (mgr.14176) 992 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:15.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:14 smithi156 bash[29655]: audit 2023-11-17T05:55:13.636010+0000 mgr.smithi006.odexej (mgr.14176) 992 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:15.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:16.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:15 smithi006 bash[26022]: cluster 2023-11-17T05:55:14.804535+0000 mgr.smithi006.odexej (mgr.14176) 993 : cluster [DBG] pgmap v696: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:16.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:15 smithi156 bash[29655]: cluster 2023-11-17T05:55:14.804535+0000 mgr.smithi006.odexej (mgr.14176) 993 : cluster [DBG] pgmap v696: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:17.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:17 smithi006 bash[26022]: cluster 2023-11-17T05:55:16.805575+0000 mgr.smithi006.odexej (mgr.14176) 994 : cluster [DBG] pgmap v697: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:17.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:17 smithi156 bash[29655]: cluster 2023-11-17T05:55:16.805575+0000 mgr.smithi006.odexej (mgr.14176) 994 : cluster [DBG] pgmap v697: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:18.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:18.377 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:18.984 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:19.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:20.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:19 smithi006 bash[26022]: audit 2023-11-17T05:55:18.358538+0000 mgr.smithi006.odexej (mgr.14176) 995 : audit [DBG] from='client.14974 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:20.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:19 smithi006 bash[26022]: cluster 2023-11-17T05:55:18.806855+0000 mgr.smithi006.odexej (mgr.14176) 996 : cluster [DBG] pgmap v698: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:19 smithi156 bash[29655]: audit 2023-11-17T05:55:18.358538+0000 mgr.smithi006.odexej (mgr.14176) 995 : audit [DBG] from='client.14974 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:20.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:19 smithi156 bash[29655]: cluster 2023-11-17T05:55:18.806855+0000 mgr.smithi006.odexej (mgr.14176) 996 : cluster [DBG] pgmap v698: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:22.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:21 smithi006 bash[26022]: cluster 2023-11-17T05:55:20.808256+0000 mgr.smithi006.odexej (mgr.14176) 997 : cluster [DBG] pgmap v699: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:22.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:21 smithi156 bash[29655]: cluster 2023-11-17T05:55:20.808256+0000 mgr.smithi006.odexej (mgr.14176) 997 : cluster [DBG] pgmap v699: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:23.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:23.156 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:23.701 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:24.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:23 smithi006 bash[26022]: cluster 2023-11-17T05:55:22.809603+0000 mgr.smithi006.odexej (mgr.14176) 998 : cluster [DBG] pgmap v700: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:24.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:23 smithi156 bash[29655]: cluster 2023-11-17T05:55:22.809603+0000 mgr.smithi006.odexej (mgr.14176) 998 : cluster [DBG] pgmap v700: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:24.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:25.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:24 smithi006 bash[26022]: audit 2023-11-17T05:55:23.146047+0000 mgr.smithi006.odexej (mgr.14176) 999 : audit [DBG] from='client.24773 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:25.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:24 smithi156 bash[29655]: audit 2023-11-17T05:55:23.146047+0000 mgr.smithi006.odexej (mgr.14176) 999 : audit [DBG] from='client.24773 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:26.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:25 smithi006 bash[26022]: cluster 2023-11-17T05:55:24.810995+0000 mgr.smithi006.odexej (mgr.14176) 1000 : cluster [DBG] pgmap v701: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:26.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:25 smithi156 bash[29655]: cluster 2023-11-17T05:55:24.810995+0000 mgr.smithi006.odexej (mgr.14176) 1000 : cluster [DBG] pgmap v701: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:27.721 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:53:18.452524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xlrxih on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:27.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:26.776649+0000 mon.smithi006 (mon.0) 908 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:55:27.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: cluster 2023-11-17T05:55:26.812077+0000 mgr.smithi006.odexej (mgr.14176) 1001 : cluster [DBG] pgmap v702: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.089889+0000 mon.smithi006 (mon.0) 909 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.404973+0000 mon.smithi006 (mon.0) 910 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.406595+0000 mon.smithi006 (mon.0) 911 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.413997+0000 mon.smithi006 (mon.0) 912 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.421041+0000 mon.smithi006 (mon.0) 913 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:55:27.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:27 smithi006 bash[26022]: audit 2023-11-17T05:55:27.435994+0000 mon.smithi006 (mon.0) 914 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:27.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:26.776649+0000 mon.smithi006 (mon.0) 908 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: cluster 2023-11-17T05:55:26.812077+0000 mgr.smithi006.odexej (mgr.14176) 1001 : cluster [DBG] pgmap v702: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.089889+0000 mon.smithi006 (mon.0) 909 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.404973+0000 mon.smithi006 (mon.0) 910 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.406595+0000 mon.smithi006 (mon.0) 911 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.413997+0000 mon.smithi006 (mon.0) 912 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.421041+0000 mon.smithi006 (mon.0) 913 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:55:27.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:27 smithi156 bash[29655]: audit 2023-11-17T05:55:27.435994+0000 mon.smithi006 (mon.0) 914 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:28.327 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:28 smithi006 bash[26022]: cluster 2023-11-17T05:55:27.414843+0000 mgr.smithi006.odexej (mgr.14176) 1002 : cluster [DBG] pgmap v703: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:28 smithi006 bash[26022]: cephadm 2023-11-17T05:55:27.442018+0000 mgr.smithi006.odexej (mgr.14176) 1003 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xdemmm on smithi156 2023-11-17T05:55:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:28 smithi006 bash[26022]: cluster 2023-11-17T05:55:27.554240+0000 mon.smithi006 (mon.0) 915 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:55:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:28 smithi006 bash[26022]: cluster 2023-11-17T05:55:27.554331+0000 mon.smithi006 (mon.0) 916 : cluster [INF] Cluster is now healthy 2023-11-17T05:55:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:28 smithi006 bash[26022]: audit 2023-11-17T05:55:27.700973+0000 mgr.smithi006.odexej (mgr.14176) 1004 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:28.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:28 smithi156 bash[29655]: cluster 2023-11-17T05:55:27.414843+0000 mgr.smithi006.odexej (mgr.14176) 1002 : cluster [DBG] pgmap v703: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:28.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:28 smithi156 bash[29655]: cephadm 2023-11-17T05:55:27.442018+0000 mgr.smithi006.odexej (mgr.14176) 1003 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.xdemmm on smithi156 2023-11-17T05:55:28.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:28 smithi156 bash[29655]: cluster 2023-11-17T05:55:27.554240+0000 mon.smithi006 (mon.0) 915 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:55:28.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:28 smithi156 bash[29655]: cluster 2023-11-17T05:55:27.554331+0000 mon.smithi006 (mon.0) 916 : cluster [INF] Cluster is now healthy 2023-11-17T05:55:28.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:28 smithi156 bash[29655]: audit 2023-11-17T05:55:27.700973+0000 mgr.smithi006.odexej (mgr.14176) 1004 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:29.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:30.695 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:30 smithi006 bash[26022]: cluster 2023-11-17T05:55:29.416056+0000 mgr.smithi006.odexej (mgr.14176) 1005 : cluster [DBG] pgmap v704: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:30.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:30 smithi156 bash[29655]: cluster 2023-11-17T05:55:29.416056+0000 mgr.smithi006.odexej (mgr.14176) 1005 : cluster [DBG] pgmap v704: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:31.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:31 smithi006 bash[26022]: audit 2023-11-17T05:55:31.175008+0000 mon.smithi006 (mon.0) 917 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xdemmm"}]: dispatch 2023-11-17T05:55:31.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:31 smithi156 bash[29655]: audit 2023-11-17T05:55:31.175008+0000 mon.smithi006 (mon.0) 917 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.xdemmm"}]: dispatch 2023-11-17T05:55:32.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:32.731 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:22.610860Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xanutw on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xanutw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xanutw\nDeploy daemon haproxy.nfs.foo.smithi156.xanutw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.766108Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.qgzvtd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-qgzvtd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.qgzvtd\nDeploy daemon haproxy.nfs.foo.smithi006.qgzvtd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:54:26.769944Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dgvrwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: cephadm 2023-11-17T05:55:31.173895+0000 mgr.smithi006.odexej (mgr.14176) 1006 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: cephadm 2023-11-17T05:55:31.174405+0000 mgr.smithi006.odexej (mgr.14176) 1007 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: cephadm 2023-11-17T05:55:31.176649+0000 mgr.smithi006.odexej (mgr.14176) 1008 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: cephadm 2023-11-17T05:55:31.180914+0000 mgr.smithi006.odexej (mgr.14176) 1009 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.dgmrlp on smithi006 2023-11-17T05:55:32.921 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:32 smithi006 bash[26022]: cluster 2023-11-17T05:55:31.416672+0000 mgr.smithi006.odexej (mgr.14176) 1010 : cluster [DBG] pgmap v705: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: cephadm 2023-11-17T05:55:31.173895+0000 mgr.smithi006.odexej (mgr.14176) 1006 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.935 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.936 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: cephadm 2023-11-17T05:55:31.174405+0000 mgr.smithi006.odexej (mgr.14176) 1007 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: cephadm 2023-11-17T05:55:31.176649+0000 mgr.smithi006.odexej (mgr.14176) 1008 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.xdemmm ... 2023-11-17T05:55:32.937 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: cephadm 2023-11-17T05:55:31.180914+0000 mgr.smithi006.odexej (mgr.14176) 1009 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.dgmrlp on smithi006 2023-11-17T05:55:32.938 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:32 smithi156 bash[29655]: cluster 2023-11-17T05:55:31.416672+0000 mgr.smithi006.odexej (mgr.14176) 1010 : cluster [DBG] pgmap v705: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:33.301 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:33.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:33 smithi006 bash[26022]: audit 2023-11-17T05:55:32.714101+0000 mgr.smithi006.odexej (mgr.14176) 1011 : audit [DBG] from='client.14986 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:33.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:33 smithi156 bash[29655]: audit 2023-11-17T05:55:32.714101+0000 mgr.smithi006.odexej (mgr.14176) 1011 : audit [DBG] from='client.14986 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:34.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:34.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:34 smithi006 bash[26022]: cluster 2023-11-17T05:55:33.417564+0000 mgr.smithi006.odexej (mgr.14176) 1012 : cluster [DBG] pgmap v706: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:34.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:34 smithi156 bash[29655]: cluster 2023-11-17T05:55:33.417564+0000 mgr.smithi006.odexej (mgr.14176) 1012 : cluster [DBG] pgmap v706: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T05:55:35.852 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:35 smithi006 bash[26022]: audit 2023-11-17T05:55:34.725620+0000 mon.smithi006 (mon.0) 918 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:35.852 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:35 smithi006 bash[26022]: audit 2023-11-17T05:55:35.354930+0000 mon.smithi006 (mon.0) 919 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.dgmrlp"}]: dispatch 2023-11-17T05:55:35.852 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:35 smithi006 bash[26022]: audit 2023-11-17T05:55:35.719459+0000 mon.smithi006 (mon.0) 920 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:36.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:35 smithi156 bash[29655]: audit 2023-11-17T05:55:34.725620+0000 mon.smithi006 (mon.0) 918 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:36.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:35 smithi156 bash[29655]: audit 2023-11-17T05:55:35.354930+0000 mon.smithi006 (mon.0) 919 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.dgmrlp"}]: dispatch 2023-11-17T05:55:36.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:35 smithi156 bash[29655]: audit 2023-11-17T05:55:35.719459+0000 mon.smithi006 (mon.0) 920 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:37.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cephadm 2023-11-17T05:55:35.354121+0000 mgr.smithi006.odexej (mgr.14176) 1013 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:55:37.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.139 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cephadm 2023-11-17T05:55:35.354482+0000 mgr.smithi006.odexej (mgr.14176) 1014 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cephadm 2023-11-17T05:55:35.356052+0000 mgr.smithi006.odexej (mgr.14176) 1015 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cephadm 2023-11-17T05:55:35.358004+0000 mgr.smithi006.odexej (mgr.14176) 1016 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cephadm 2023-11-17T05:55:35.359819+0000 mgr.smithi006.odexej (mgr.14176) 1017 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cluster 2023-11-17T05:55:35.361276+0000 mgr.smithi006.odexej (mgr.14176) 1018 : cluster [DBG] pgmap v707: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:55:37.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:36 smithi006 bash[26022]: cluster 2023-11-17T05:55:35.721521+0000 mon.smithi006 (mon.0) 921 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cephadm 2023-11-17T05:55:35.354121+0000 mgr.smithi006.odexej (mgr.14176) 1013 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cephadm 2023-11-17T05:55:35.354482+0000 mgr.smithi006.odexej (mgr.14176) 1014 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cephadm 2023-11-17T05:55:35.356052+0000 mgr.smithi006.odexej (mgr.14176) 1015 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dgmrlp ... 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cephadm 2023-11-17T05:55:35.358004+0000 mgr.smithi006.odexej (mgr.14176) 1016 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cephadm 2023-11-17T05:55:35.359819+0000 mgr.smithi006.odexej (mgr.14176) 1017 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:55:37.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cluster 2023-11-17T05:55:35.361276+0000 mgr.smithi006.odexej (mgr.14176) 1018 : cluster [DBG] pgmap v707: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:55:37.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:36 smithi156 bash[29655]: cluster 2023-11-17T05:55:35.721521+0000 mon.smithi006 (mon.0) 921 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:55:37.604 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:37.604 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.355903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\nDeploy daemon haproxy.nfs.foo.smithi006.dgmrlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.357888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:38.232 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:39.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:38 smithi006 bash[26022]: cluster 2023-11-17T05:55:37.361992+0000 mgr.smithi006.odexej (mgr.14176) 1019 : cluster [DBG] pgmap v708: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:55:39.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:38 smithi006 bash[26022]: audit 2023-11-17T05:55:37.585628+0000 mgr.smithi006.odexej (mgr.14176) 1020 : audit [DBG] from='client.14990 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:39.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:38 smithi156 bash[29655]: cluster 2023-11-17T05:55:37.361992+0000 mgr.smithi006.odexej (mgr.14176) 1019 : cluster [DBG] pgmap v708: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2023-11-17T05:55:39.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:38 smithi156 bash[29655]: audit 2023-11-17T05:55:37.585628+0000 mgr.smithi006.odexej (mgr.14176) 1020 : audit [DBG] from='client.14990 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:39.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:41.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:40 smithi006 bash[26022]: cluster 2023-11-17T05:55:39.362820+0000 mgr.smithi006.odexej (mgr.14176) 1021 : cluster [DBG] pgmap v709: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:41.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:40 smithi156 bash[29655]: cluster 2023-11-17T05:55:39.362820+0000 mgr.smithi006.odexej (mgr.14176) 1021 : cluster [DBG] pgmap v709: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.355903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\nDeploy daemon haproxy.nfs.foo.smithi006.dgmrlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.357888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:42.790 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:43.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:42 smithi006 bash[26022]: cluster 2023-11-17T05:55:41.363916+0000 mgr.smithi006.odexej (mgr.14176) 1022 : cluster [DBG] pgmap v710: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:43.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:42 smithi156 bash[29655]: cluster 2023-11-17T05:55:41.363916+0000 mgr.smithi006.odexej (mgr.14176) 1022 : cluster [DBG] pgmap v710: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:43.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:43.934 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:43 smithi156 bash[29655]: audit 2023-11-17T05:55:42.194935+0000 mgr.smithi006.odexej (mgr.14176) 1023 : audit [DBG] from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:44.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:43 smithi006 bash[26022]: audit 2023-11-17T05:55:42.194935+0000 mgr.smithi006.odexej (mgr.14176) 1023 : audit [DBG] from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:45.102 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:44 smithi006 bash[26022]: cluster 2023-11-17T05:55:43.365071+0000 mgr.smithi006.odexej (mgr.14176) 1024 : cluster [DBG] pgmap v711: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T05:55:45.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:44 smithi156 bash[29655]: cluster 2023-11-17T05:55:43.365071+0000 mgr.smithi006.odexej (mgr.14176) 1024 : cluster [DBG] pgmap v711: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T05:55:46.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:54.944 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.355903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\nDeploy daemon haproxy.nfs.foo.smithi006.dgmrlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.357888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:54.945 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:46 smithi006 bash[26022]: cluster 2023-11-17T05:55:45.366383+0000 mgr.smithi006.odexej (mgr.14176) 1025 : cluster [DBG] pgmap v712: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.946 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:46 smithi156 bash[29655]: cluster 2023-11-17T05:55:45.366383+0000 mgr.smithi006.odexej (mgr.14176) 1025 : cluster [DBG] pgmap v712: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.946 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:47 smithi156 bash[29655]: audit 2023-11-17T05:55:46.787990+0000 mgr.smithi006.odexej (mgr.14176) 1026 : audit [DBG] from='client.14998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:54.946 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:48 smithi156 bash[29655]: cluster 2023-11-17T05:55:47.367495+0000 mgr.smithi006.odexej (mgr.14176) 1027 : cluster [DBG] pgmap v713: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.947 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:50 smithi156 bash[29655]: cluster 2023-11-17T05:55:49.368993+0000 mgr.smithi006.odexej (mgr.14176) 1028 : cluster [DBG] pgmap v714: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.947 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:50 smithi156 bash[29655]: audit 2023-11-17T05:55:49.726111+0000 mon.smithi006 (mon.0) 922 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:54.947 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:52 smithi156 bash[29655]: cluster 2023-11-17T05:55:51.369948+0000 mgr.smithi006.odexej (mgr.14176) 1029 : cluster [DBG] pgmap v715: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.947 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:54.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:47 smithi006 bash[26022]: audit 2023-11-17T05:55:46.787990+0000 mgr.smithi006.odexej (mgr.14176) 1026 : audit [DBG] from='client.14998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:55:54.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:48 smithi006 bash[26022]: cluster 2023-11-17T05:55:47.367495+0000 mgr.smithi006.odexej (mgr.14176) 1027 : cluster [DBG] pgmap v713: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:50 smithi006 bash[26022]: cluster 2023-11-17T05:55:49.368993+0000 mgr.smithi006.odexej (mgr.14176) 1028 : cluster [DBG] pgmap v714: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:54.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:50 smithi006 bash[26022]: audit 2023-11-17T05:55:49.726111+0000 mon.smithi006 (mon.0) 922 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:55:54.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:52 smithi006 bash[26022]: cluster 2023-11-17T05:55:51.369948+0000 mgr.smithi006.odexej (mgr.14176) 1029 : cluster [DBG] pgmap v715: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:55.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:54 smithi006 bash[26022]: cluster 2023-11-17T05:55:53.371051+0000 mgr.smithi006.odexej (mgr.14176) 1030 : cluster [DBG] pgmap v716: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:55.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:54 smithi156 bash[29655]: cluster 2023-11-17T05:55:53.371051+0000 mgr.smithi006.odexej (mgr.14176) 1030 : cluster [DBG] pgmap v716: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:55:55.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:55:57.009 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:56 smithi006 bash[26022]: cluster 2023-11-17T05:55:55.372428+0000 mgr.smithi006.odexej (mgr.14176) 1031 : cluster [DBG] pgmap v717: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:57.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:56 smithi156 bash[29655]: cluster 2023-11-17T05:55:55.372428+0000 mgr.smithi006.odexej (mgr.14176) 1031 : cluster [DBG] pgmap v717: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:58.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:58 smithi006 bash[26022]: cluster 2023-11-17T05:55:57.373787+0000 mgr.smithi006.odexej (mgr.14176) 1032 : cluster [DBG] pgmap v718: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:58.988 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:55:58.988 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.355903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\nDeploy daemon haproxy.nfs.foo.smithi006.dgmrlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.357888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:55:59.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:58 smithi156 bash[29655]: cluster 2023-11-17T05:55:57.373787+0000 mgr.smithi006.odexej (mgr.14176) 1032 : cluster [DBG] pgmap v718: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:55:59.586 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:55:59.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:55:59 smithi006 bash[26022]: audit 2023-11-17T05:55:58.973307+0000 mgr.smithi006.odexej (mgr.14176) 1033 : audit [DBG] from='client.15002 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:00.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:55:59 smithi156 bash[29655]: audit 2023-11-17T05:55:58.973307+0000 mgr.smithi006.odexej (mgr.14176) 1033 : audit [DBG] from='client.15002 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:00.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:56:00.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:00 smithi006 bash[26022]: cluster 2023-11-17T05:55:59.375283+0000 mgr.smithi006.odexej (mgr.14176) 1034 : cluster [DBG] pgmap v719: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:01.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:00 smithi156 bash[29655]: cluster 2023-11-17T05:55:59.375283+0000 mgr.smithi006.odexej (mgr.14176) 1034 : cluster [DBG] pgmap v719: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:03.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:02 smithi006 bash[26022]: cluster 2023-11-17T05:56:01.376480+0000 mgr.smithi006.odexej (mgr.14176) 1035 : cluster [DBG] pgmap v720: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:49.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:56:49.935 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:54:26.772094Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.jwojjj on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:31.176421Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.xdemmm on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-xdemmm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.xdemmm\nDeploy daemon haproxy.nfs.foo.smithi156.xdemmm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.355903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dgmrlp on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dgmrlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dgmrlp\nDeploy daemon haproxy.nfs.foo.smithi006.dgmrlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:55:35.357888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.reuuql on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:56:49.965 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:56:49.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:02 smithi156 bash[29655]: cluster 2023-11-17T05:56:01.376480+0000 mgr.smithi006.odexej (mgr.14176) 1035 : cluster [DBG] pgmap v720: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.001 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:04 smithi156 bash[29655]: cluster 2023-11-17T05:56:03.377962+0000 mgr.smithi006.odexej (mgr.14176) 1036 : cluster [DBG] pgmap v721: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:04 smithi156 bash[29655]: audit 2023-11-17T05:56:03.775901+0000 mgr.smithi006.odexej (mgr.14176) 1037 : audit [DBG] from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:06 smithi156 bash[29655]: cluster 2023-11-17T05:56:05.379398+0000 mgr.smithi006.odexej (mgr.14176) 1038 : cluster [DBG] pgmap v722: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:08 smithi156 bash[29655]: cluster 2023-11-17T05:56:07.380580+0000 mgr.smithi006.odexej (mgr.14176) 1039 : cluster [DBG] pgmap v723: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:10 smithi156 bash[29655]: cluster 2023-11-17T05:56:09.382161+0000 mgr.smithi006.odexej (mgr.14176) 1040 : cluster [DBG] pgmap v724: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:12 smithi156 bash[29655]: cluster 2023-11-17T05:56:11.383370+0000 mgr.smithi006.odexej (mgr.14176) 1041 : cluster [DBG] pgmap v725: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:14 smithi156 bash[29655]: cluster 2023-11-17T05:56:13.384645+0000 mgr.smithi006.odexej (mgr.14176) 1042 : cluster [DBG] pgmap v726: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:16 smithi156 bash[29655]: cluster 2023-11-17T05:56:15.386072+0000 mgr.smithi006.odexej (mgr.14176) 1043 : cluster [DBG] pgmap v727: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:18 smithi156 bash[29655]: cluster 2023-11-17T05:56:17.387113+0000 mgr.smithi006.odexej (mgr.14176) 1044 : cluster [DBG] pgmap v728: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:20 smithi156 bash[29655]: cluster 2023-11-17T05:56:19.388483+0000 mgr.smithi006.odexej (mgr.14176) 1045 : cluster [DBG] pgmap v729: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:22 smithi156 bash[29655]: cluster 2023-11-17T05:56:21.389617+0000 mgr.smithi006.odexej (mgr.14176) 1046 : cluster [DBG] pgmap v730: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:24 smithi156 bash[29655]: cluster 2023-11-17T05:56:23.390777+0000 mgr.smithi006.odexej (mgr.14176) 1047 : cluster [DBG] pgmap v731: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.002 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:26 smithi156 bash[29655]: cluster 2023-11-17T05:56:25.392160+0000 mgr.smithi006.odexej (mgr.14176) 1048 : cluster [DBG] pgmap v732: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:28 smithi156 bash[29655]: cluster 2023-11-17T05:56:27.393167+0000 mgr.smithi006.odexej (mgr.14176) 1049 : cluster [DBG] pgmap v733: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:30 smithi156 bash[29655]: cluster 2023-11-17T05:56:29.394527+0000 mgr.smithi006.odexej (mgr.14176) 1050 : cluster [DBG] pgmap v734: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:32 smithi156 bash[29655]: cluster 2023-11-17T05:56:31.395787+0000 mgr.smithi006.odexej (mgr.14176) 1051 : cluster [DBG] pgmap v735: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:34 smithi156 bash[29655]: cluster 2023-11-17T05:56:33.397097+0000 mgr.smithi006.odexej (mgr.14176) 1052 : cluster [DBG] pgmap v736: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.363907+0000 mon.smithi006 (mon.0) 923 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.690880+0000 mon.smithi006 (mon.0) 924 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.692579+0000 mon.smithi006 (mon.0) 925 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.701359+0000 mon.smithi006 (mon.0) 926 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.710503+0000 mon.smithi006 (mon.0) 927 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:35 smithi156 bash[29655]: audit 2023-11-17T05:56:35.726240+0000 mon.smithi006 (mon.0) 928 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:36 smithi156 bash[29655]: cluster 2023-11-17T05:56:35.398498+0000 mgr.smithi006.odexej (mgr.14176) 1053 : cluster [DBG] pgmap v737: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.003 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:36 smithi156 bash[29655]: cluster 2023-11-17T05:56:35.702388+0000 mgr.smithi006.odexej (mgr.14176) 1054 : cluster [DBG] pgmap v738: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:36 smithi156 bash[29655]: cephadm 2023-11-17T05:56:35.729975+0000 mgr.smithi006.odexej (mgr.14176) 1055 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.hzgjze on smithi156 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:36 smithi156 bash[29655]: cluster 2023-11-17T05:56:35.866246+0000 mon.smithi006 (mon.0) 929 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:36 smithi156 bash[29655]: cluster 2023-11-17T05:56:35.866318+0000 mon.smithi006 (mon.0) 930 : cluster [INF] Cluster is now healthy 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:38 smithi156 bash[29655]: cluster 2023-11-17T05:56:37.703158+0000 mgr.smithi006.odexej (mgr.14176) 1056 : cluster [DBG] pgmap v739: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:39 smithi156 bash[29655]: audit 2023-11-17T05:56:39.590332+0000 mon.smithi006 (mon.0) 931 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.hzgjze"}]: dispatch 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: cephadm 2023-11-17T05:56:39.589038+0000 mgr.smithi006.odexej (mgr.14176) 1057 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.004 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.005 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.073 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.073 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.073 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.073 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.073 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.074 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: cephadm 2023-11-17T05:56:39.589568+0000 mgr.smithi006.odexej (mgr.14176) 1058 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.075 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: cephadm 2023-11-17T05:56:39.592249+0000 mgr.smithi006.odexej (mgr.14176) 1059 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.076 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: cephadm 2023-11-17T05:56:39.596925+0000 mgr.smithi006.odexej (mgr.14176) 1060 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.ditjtc on smithi006 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:40 smithi156 bash[29655]: cluster 2023-11-17T05:56:39.704346+0000 mgr.smithi006.odexej (mgr.14176) 1061 : cluster [DBG] pgmap v740: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:42 smithi156 bash[29655]: cluster 2023-11-17T05:56:41.705147+0000 mgr.smithi006.odexej (mgr.14176) 1062 : cluster [DBG] pgmap v741: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:43 smithi156 bash[29655]: audit 2023-11-17T05:56:43.319560+0000 mon.smithi006 (mon.0) 932 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.ditjtc"}]: dispatch 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cephadm 2023-11-17T05:56:43.318512+0000 mgr.smithi006.odexej (mgr.14176) 1063 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.077 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.078 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cephadm 2023-11-17T05:56:43.318959+0000 mgr.smithi006.odexej (mgr.14176) 1064 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cephadm 2023-11-17T05:56:43.321052+0000 mgr.smithi006.odexej (mgr.14176) 1065 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.079 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cephadm 2023-11-17T05:56:43.325373+0000 mgr.smithi006.odexej (mgr.14176) 1066 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cephadm 2023-11-17T05:56:43.328388+0000 mgr.smithi006.odexej (mgr.14176) 1067 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cluster 2023-11-17T05:56:43.329680+0000 mgr.smithi006.odexej (mgr.14176) 1068 : cluster [DBG] pgmap v742: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:56:50.080 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:44 smithi156 bash[29655]: cluster 2023-11-17T05:56:43.588914+0000 mon.smithi006 (mon.0) 933 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:56:50.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:46 smithi156 bash[29655]: cluster 2023-11-17T05:56:45.331210+0000 mgr.smithi006.odexej (mgr.14176) 1069 : cluster [DBG] pgmap v743: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:56:50.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:46 smithi156 bash[29655]: audit 2023-11-17T05:56:45.774932+0000 mon.smithi006 (mon.0) 934 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.081 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:48 smithi156 bash[29655]: cluster 2023-11-17T05:56:47.332314+0000 mgr.smithi006.odexej (mgr.14176) 1070 : cluster [DBG] pgmap v744: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:50.081 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:04 smithi006 bash[26022]: cluster 2023-11-17T05:56:03.377962+0000 mgr.smithi006.odexej (mgr.14176) 1036 : cluster [DBG] pgmap v721: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:04 smithi006 bash[26022]: audit 2023-11-17T05:56:03.775901+0000 mgr.smithi006.odexej (mgr.14176) 1037 : audit [DBG] from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:06 smithi006 bash[26022]: cluster 2023-11-17T05:56:05.379398+0000 mgr.smithi006.odexej (mgr.14176) 1038 : cluster [DBG] pgmap v722: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:08 smithi006 bash[26022]: cluster 2023-11-17T05:56:07.380580+0000 mgr.smithi006.odexej (mgr.14176) 1039 : cluster [DBG] pgmap v723: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:10 smithi006 bash[26022]: cluster 2023-11-17T05:56:09.382161+0000 mgr.smithi006.odexej (mgr.14176) 1040 : cluster [DBG] pgmap v724: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:12 smithi006 bash[26022]: cluster 2023-11-17T05:56:11.383370+0000 mgr.smithi006.odexej (mgr.14176) 1041 : cluster [DBG] pgmap v725: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:14 smithi006 bash[26022]: cluster 2023-11-17T05:56:13.384645+0000 mgr.smithi006.odexej (mgr.14176) 1042 : cluster [DBG] pgmap v726: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:16 smithi006 bash[26022]: cluster 2023-11-17T05:56:15.386072+0000 mgr.smithi006.odexej (mgr.14176) 1043 : cluster [DBG] pgmap v727: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:18 smithi006 bash[26022]: cluster 2023-11-17T05:56:17.387113+0000 mgr.smithi006.odexej (mgr.14176) 1044 : cluster [DBG] pgmap v728: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:20 smithi006 bash[26022]: cluster 2023-11-17T05:56:19.388483+0000 mgr.smithi006.odexej (mgr.14176) 1045 : cluster [DBG] pgmap v729: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:22 smithi006 bash[26022]: cluster 2023-11-17T05:56:21.389617+0000 mgr.smithi006.odexej (mgr.14176) 1046 : cluster [DBG] pgmap v730: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:24 smithi006 bash[26022]: cluster 2023-11-17T05:56:23.390777+0000 mgr.smithi006.odexej (mgr.14176) 1047 : cluster [DBG] pgmap v731: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:26 smithi006 bash[26022]: cluster 2023-11-17T05:56:25.392160+0000 mgr.smithi006.odexej (mgr.14176) 1048 : cluster [DBG] pgmap v732: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:28 smithi006 bash[26022]: cluster 2023-11-17T05:56:27.393167+0000 mgr.smithi006.odexej (mgr.14176) 1049 : cluster [DBG] pgmap v733: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:30 smithi006 bash[26022]: cluster 2023-11-17T05:56:29.394527+0000 mgr.smithi006.odexej (mgr.14176) 1050 : cluster [DBG] pgmap v734: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:32 smithi006 bash[26022]: cluster 2023-11-17T05:56:31.395787+0000 mgr.smithi006.odexej (mgr.14176) 1051 : cluster [DBG] pgmap v735: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:34 smithi006 bash[26022]: cluster 2023-11-17T05:56:33.397097+0000 mgr.smithi006.odexej (mgr.14176) 1052 : cluster [DBG] pgmap v736: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.363907+0000 mon.smithi006 (mon.0) 923 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.690880+0000 mon.smithi006 (mon.0) 924 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.692579+0000 mon.smithi006 (mon.0) 925 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.701359+0000 mon.smithi006 (mon.0) 926 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.710503+0000 mon.smithi006 (mon.0) 927 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:35 smithi006 bash[26022]: audit 2023-11-17T05:56:35.726240+0000 mon.smithi006 (mon.0) 928 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:36 smithi006 bash[26022]: cluster 2023-11-17T05:56:35.398498+0000 mgr.smithi006.odexej (mgr.14176) 1053 : cluster [DBG] pgmap v737: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:36 smithi006 bash[26022]: cluster 2023-11-17T05:56:35.702388+0000 mgr.smithi006.odexej (mgr.14176) 1054 : cluster [DBG] pgmap v738: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:36 smithi006 bash[26022]: cephadm 2023-11-17T05:56:35.729975+0000 mgr.smithi006.odexej (mgr.14176) 1055 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.hzgjze on smithi156 2023-11-17T05:56:50.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:36 smithi006 bash[26022]: cluster 2023-11-17T05:56:35.866246+0000 mon.smithi006 (mon.0) 929 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:36 smithi006 bash[26022]: cluster 2023-11-17T05:56:35.866318+0000 mon.smithi006 (mon.0) 930 : cluster [INF] Cluster is now healthy 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:38 smithi006 bash[26022]: cluster 2023-11-17T05:56:37.703158+0000 mgr.smithi006.odexej (mgr.14176) 1056 : cluster [DBG] pgmap v739: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:39 smithi006 bash[26022]: audit 2023-11-17T05:56:39.590332+0000 mon.smithi006 (mon.0) 931 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.hzgjze"}]: dispatch 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: cephadm 2023-11-17T05:56:39.589038+0000 mgr.smithi006.odexej (mgr.14176) 1057 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: cephadm 2023-11-17T05:56:39.589568+0000 mgr.smithi006.odexej (mgr.14176) 1058 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: cephadm 2023-11-17T05:56:39.592249+0000 mgr.smithi006.odexej (mgr.14176) 1059 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze 2023-11-17T05:56:50.086 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.hzgjze ... 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: cephadm 2023-11-17T05:56:39.596925+0000 mgr.smithi006.odexej (mgr.14176) 1060 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.ditjtc on smithi006 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:40 smithi006 bash[26022]: cluster 2023-11-17T05:56:39.704346+0000 mgr.smithi006.odexej (mgr.14176) 1061 : cluster [DBG] pgmap v740: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:42 smithi006 bash[26022]: cluster 2023-11-17T05:56:41.705147+0000 mgr.smithi006.odexej (mgr.14176) 1062 : cluster [DBG] pgmap v741: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:43 smithi006 bash[26022]: audit 2023-11-17T05:56:43.319560+0000 mon.smithi006 (mon.0) 932 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.ditjtc"}]: dispatch 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cephadm 2023-11-17T05:56:43.318512+0000 mgr.smithi006.odexej (mgr.14176) 1063 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:56:50.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cephadm 2023-11-17T05:56:43.318959+0000 mgr.smithi006.odexej (mgr.14176) 1064 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cephadm 2023-11-17T05:56:43.321052+0000 mgr.smithi006.odexej (mgr.14176) 1065 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ditjtc ... 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cephadm 2023-11-17T05:56:43.325373+0000 mgr.smithi006.odexej (mgr.14176) 1066 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cephadm 2023-11-17T05:56:43.328388+0000 mgr.smithi006.odexej (mgr.14176) 1067 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:56:50.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cluster 2023-11-17T05:56:43.329680+0000 mgr.smithi006.odexej (mgr.14176) 1068 : cluster [DBG] pgmap v742: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:56:50.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:44 smithi006 bash[26022]: cluster 2023-11-17T05:56:43.588914+0000 mon.smithi006 (mon.0) 933 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:56:50.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:46 smithi006 bash[26022]: cluster 2023-11-17T05:56:45.331210+0000 mgr.smithi006.odexej (mgr.14176) 1069 : cluster [DBG] pgmap v743: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T05:56:50.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:46 smithi006 bash[26022]: audit 2023-11-17T05:56:45.774932+0000 mon.smithi006 (mon.0) 934 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:56:50.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:48 smithi006 bash[26022]: cluster 2023-11-17T05:56:47.332314+0000 mgr.smithi006.odexej (mgr.14176) 1070 : cluster [DBG] pgmap v744: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:50.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:56:51.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:50 smithi006 bash[26022]: cluster 2023-11-17T05:56:49.333635+0000 mgr.smithi006.odexej (mgr.14176) 1071 : cluster [DBG] pgmap v745: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:51.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:50 smithi156 bash[29655]: cluster 2023-11-17T05:56:49.333635+0000 mgr.smithi006.odexej (mgr.14176) 1071 : cluster [DBG] pgmap v745: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:53.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:52 smithi006 bash[26022]: cluster 2023-11-17T05:56:51.334630+0000 mgr.smithi006.odexej (mgr.14176) 1072 : cluster [DBG] pgmap v746: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:53.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:52 smithi156 bash[29655]: cluster 2023-11-17T05:56:51.334630+0000 mgr.smithi006.odexej (mgr.14176) 1072 : cluster [DBG] pgmap v746: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:56:53.932 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:56:53.932 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:56:54.551 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:56:55.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:54 smithi006 bash[26022]: cluster 2023-11-17T05:56:53.335812+0000 mgr.smithi006.odexej (mgr.14176) 1073 : cluster [DBG] pgmap v747: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T05:56:55.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:54 smithi006 bash[26022]: audit 2023-11-17T05:56:53.915429+0000 mgr.smithi006.odexej (mgr.14176) 1074 : audit [DBG] from='client.15010 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:55.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:54 smithi156 bash[29655]: cluster 2023-11-17T05:56:53.335812+0000 mgr.smithi006.odexej (mgr.14176) 1073 : cluster [DBG] pgmap v747: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T05:56:55.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:54 smithi156 bash[29655]: audit 2023-11-17T05:56:53.915429+0000 mgr.smithi006.odexej (mgr.14176) 1074 : audit [DBG] from='client.15010 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:56:55.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:56:56.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:56 smithi006 bash[26022]: cluster 2023-11-17T05:56:55.337154+0000 mgr.smithi006.odexej (mgr.14176) 1075 : cluster [DBG] pgmap v748: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:57.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:56 smithi156 bash[29655]: cluster 2023-11-17T05:56:55.337154+0000 mgr.smithi006.odexej (mgr.14176) 1075 : cluster [DBG] pgmap v748: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:58.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:56:58.506 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:56:58.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:58 smithi006 bash[26022]: cluster 2023-11-17T05:56:57.338197+0000 mgr.smithi006.odexej (mgr.14176) 1076 : cluster [DBG] pgmap v749: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:56:59.087 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:56:59.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:58 smithi156 bash[29655]: cluster 2023-11-17T05:56:57.338197+0000 mgr.smithi006.odexej (mgr.14176) 1076 : cluster [DBG] pgmap v749: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:00.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:00.102 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:56:59 smithi006 bash[26022]: audit 2023-11-17T05:56:58.487853+0000 mgr.smithi006.odexej (mgr.14176) 1077 : audit [DBG] from='client.15014 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:00.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:56:59 smithi156 bash[29655]: audit 2023-11-17T05:56:58.487853+0000 mgr.smithi006.odexej (mgr.14176) 1077 : audit [DBG] from='client.15014 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:01.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:00 smithi006 bash[26022]: cluster 2023-11-17T05:56:59.339562+0000 mgr.smithi006.odexej (mgr.14176) 1078 : cluster [DBG] pgmap v750: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:00 smithi156 bash[29655]: cluster 2023-11-17T05:56:59.339562+0000 mgr.smithi006.odexej (mgr.14176) 1078 : cluster [DBG] pgmap v750: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:03.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:02 smithi006 bash[26022]: cluster 2023-11-17T05:57:01.340793+0000 mgr.smithi006.odexej (mgr.14176) 1079 : cluster [DBG] pgmap v751: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:03.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:02 smithi156 bash[29655]: cluster 2023-11-17T05:57:01.340793+0000 mgr.smithi006.odexej (mgr.14176) 1079 : cluster [DBG] pgmap v751: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:03.187 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:03.187 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:03.787 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:04.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:05.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:04 smithi006 bash[26022]: audit 2023-11-17T05:57:03.170785+0000 mgr.smithi006.odexej (mgr.14176) 1080 : audit [DBG] from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:05.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:04 smithi006 bash[26022]: cluster 2023-11-17T05:57:03.341964+0000 mgr.smithi006.odexej (mgr.14176) 1081 : cluster [DBG] pgmap v752: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:05.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:04 smithi156 bash[29655]: audit 2023-11-17T05:57:03.170785+0000 mgr.smithi006.odexej (mgr.14176) 1080 : audit [DBG] from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:05.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:04 smithi156 bash[29655]: cluster 2023-11-17T05:57:03.341964+0000 mgr.smithi006.odexej (mgr.14176) 1081 : cluster [DBG] pgmap v752: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:07.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:06 smithi006 bash[26022]: cluster 2023-11-17T05:57:05.343290+0000 mgr.smithi006.odexej (mgr.14176) 1082 : cluster [DBG] pgmap v753: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:07.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:06 smithi156 bash[29655]: cluster 2023-11-17T05:57:05.343290+0000 mgr.smithi006.odexej (mgr.14176) 1082 : cluster [DBG] pgmap v753: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:07.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:07.858 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:08.400 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:09.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:08 smithi006 bash[26022]: cluster 2023-11-17T05:57:07.344447+0000 mgr.smithi006.odexej (mgr.14176) 1083 : cluster [DBG] pgmap v754: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:09.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:08 smithi006 bash[26022]: audit 2023-11-17T05:57:07.847049+0000 mgr.smithi006.odexej (mgr.14176) 1084 : audit [DBG] from='client.15022 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:09.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:08 smithi156 bash[29655]: cluster 2023-11-17T05:57:07.344447+0000 mgr.smithi006.odexej (mgr.14176) 1083 : cluster [DBG] pgmap v754: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:09.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:08 smithi156 bash[29655]: audit 2023-11-17T05:57:07.847049+0000 mgr.smithi006.odexej (mgr.14176) 1084 : audit [DBG] from='client.15022 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:09.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:11.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:10 smithi006 bash[26022]: cluster 2023-11-17T05:57:09.345814+0000 mgr.smithi006.odexej (mgr.14176) 1085 : cluster [DBG] pgmap v755: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:11.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:10 smithi156 bash[29655]: cluster 2023-11-17T05:57:09.345814+0000 mgr.smithi006.odexej (mgr.14176) 1085 : cluster [DBG] pgmap v755: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:12.427 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:12.427 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:12.972 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:12 smithi006 bash[26022]: cluster 2023-11-17T05:57:11.346936+0000 mgr.smithi006.odexej (mgr.14176) 1086 : cluster [DBG] pgmap v756: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:12 smithi156 bash[29655]: cluster 2023-11-17T05:57:11.346936+0000 mgr.smithi006.odexej (mgr.14176) 1086 : cluster [DBG] pgmap v756: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:13.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:14.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:13 smithi006 bash[26022]: audit 2023-11-17T05:57:12.414245+0000 mgr.smithi006.odexej (mgr.14176) 1087 : audit [DBG] from='client.15026 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:14.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:13 smithi156 bash[29655]: audit 2023-11-17T05:57:12.414245+0000 mgr.smithi006.odexej (mgr.14176) 1087 : audit [DBG] from='client.15026 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:15.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:14 smithi006 bash[26022]: cluster 2023-11-17T05:57:13.348261+0000 mgr.smithi006.odexej (mgr.14176) 1088 : cluster [DBG] pgmap v757: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:15.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:14 smithi156 bash[29655]: cluster 2023-11-17T05:57:13.348261+0000 mgr.smithi006.odexej (mgr.14176) 1088 : cluster [DBG] pgmap v757: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:16.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:16.958 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:17.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:16 smithi006 bash[26022]: cluster 2023-11-17T05:57:15.349715+0000 mgr.smithi006.odexej (mgr.14176) 1089 : cluster [DBG] pgmap v758: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:17.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:16 smithi156 bash[29655]: cluster 2023-11-17T05:57:15.349715+0000 mgr.smithi006.odexej (mgr.14176) 1089 : cluster [DBG] pgmap v758: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:17.621 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:18.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:17 smithi006 bash[26022]: audit 2023-11-17T05:57:16.947406+0000 mgr.smithi006.odexej (mgr.14176) 1090 : audit [DBG] from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:18.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:17 smithi156 bash[29655]: audit 2023-11-17T05:57:16.947406+0000 mgr.smithi006.odexej (mgr.14176) 1090 : audit [DBG] from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:18.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:19.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:18 smithi006 bash[26022]: cluster 2023-11-17T05:57:17.350767+0000 mgr.smithi006.odexej (mgr.14176) 1091 : cluster [DBG] pgmap v759: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:19.192 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:18 smithi156 bash[29655]: cluster 2023-11-17T05:57:17.350767+0000 mgr.smithi006.odexej (mgr.14176) 1091 : cluster [DBG] pgmap v759: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:21.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:20 smithi006 bash[26022]: cluster 2023-11-17T05:57:19.352136+0000 mgr.smithi006.odexej (mgr.14176) 1092 : cluster [DBG] pgmap v760: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:21.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:20 smithi156 bash[29655]: cluster 2023-11-17T05:57:19.352136+0000 mgr.smithi006.odexej (mgr.14176) 1092 : cluster [DBG] pgmap v760: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:21.670 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:21.670 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:22.347 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:23.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:22 smithi006 bash[26022]: cluster 2023-11-17T05:57:21.352918+0000 mgr.smithi006.odexej (mgr.14176) 1093 : cluster [DBG] pgmap v761: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:23.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:22 smithi006 bash[26022]: audit 2023-11-17T05:57:21.654552+0000 mgr.smithi006.odexej (mgr.14176) 1094 : audit [DBG] from='client.15034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:23.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:22 smithi156 bash[29655]: cluster 2023-11-17T05:57:21.352918+0000 mgr.smithi006.odexej (mgr.14176) 1093 : cluster [DBG] pgmap v761: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:23.187 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:22 smithi156 bash[29655]: audit 2023-11-17T05:57:21.654552+0000 mgr.smithi006.odexej (mgr.14176) 1094 : audit [DBG] from='client.15034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:23.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:25.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:24 smithi006 bash[26022]: cluster 2023-11-17T05:57:23.354039+0000 mgr.smithi006.odexej (mgr.14176) 1095 : cluster [DBG] pgmap v762: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:25.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:24 smithi156 bash[29655]: cluster 2023-11-17T05:57:23.354039+0000 mgr.smithi006.odexej (mgr.14176) 1095 : cluster [DBG] pgmap v762: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:26.575 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:26.575 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:27.148 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:27.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:26 smithi156 bash[29655]: cluster 2023-11-17T05:57:25.355392+0000 mgr.smithi006.odexej (mgr.14176) 1096 : cluster [DBG] pgmap v763: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:27.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:26 smithi006 bash[26022]: cluster 2023-11-17T05:57:25.355392+0000 mgr.smithi006.odexej (mgr.14176) 1096 : cluster [DBG] pgmap v763: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:28.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:28.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:27 smithi006 bash[26022]: audit 2023-11-17T05:57:26.556878+0000 mgr.smithi006.odexej (mgr.14176) 1097 : audit [DBG] from='client.15038 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:28.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:27 smithi156 bash[29655]: audit 2023-11-17T05:57:26.556878+0000 mgr.smithi006.odexej (mgr.14176) 1097 : audit [DBG] from='client.15038 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:29.272 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:28 smithi006 bash[26022]: cluster 2023-11-17T05:57:27.356441+0000 mgr.smithi006.odexej (mgr.14176) 1098 : cluster [DBG] pgmap v764: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:28 smithi156 bash[29655]: cluster 2023-11-17T05:57:27.356441+0000 mgr.smithi006.odexej (mgr.14176) 1098 : cluster [DBG] pgmap v764: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:31.041 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:31.041 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:31.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:30 smithi006 bash[26022]: cluster 2023-11-17T05:57:29.357770+0000 mgr.smithi006.odexej (mgr.14176) 1099 : cluster [DBG] pgmap v765: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:31.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:30 smithi156 bash[29655]: cluster 2023-11-17T05:57:29.357770+0000 mgr.smithi006.odexej (mgr.14176) 1099 : cluster [DBG] pgmap v765: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:31.641 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:32.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:31 smithi006 bash[26022]: audit 2023-11-17T05:57:31.031219+0000 mgr.smithi006.odexej (mgr.14176) 1100 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:32.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:31 smithi156 bash[29655]: audit 2023-11-17T05:57:31.031219+0000 mgr.smithi006.odexej (mgr.14176) 1100 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:32.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:33.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:32 smithi006 bash[26022]: cluster 2023-11-17T05:57:31.358844+0000 mgr.smithi006.odexej (mgr.14176) 1101 : cluster [DBG] pgmap v766: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:33.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:32 smithi156 bash[29655]: cluster 2023-11-17T05:57:31.358844+0000 mgr.smithi006.odexej (mgr.14176) 1101 : cluster [DBG] pgmap v766: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:35.261 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:34 smithi006 bash[26022]: cluster 2023-11-17T05:57:33.360035+0000 mgr.smithi006.odexej (mgr.14176) 1102 : cluster [DBG] pgmap v767: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:35.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:34 smithi156 bash[29655]: cluster 2023-11-17T05:57:33.360035+0000 mgr.smithi006.odexej (mgr.14176) 1102 : cluster [DBG] pgmap v767: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:35.702 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:36.316 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:37.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:37.330 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:36 smithi006 bash[26022]: cluster 2023-11-17T05:57:35.361522+0000 mgr.smithi006.odexej (mgr.14176) 1103 : cluster [DBG] pgmap v768: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:37.330 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:36 smithi006 bash[26022]: audit 2023-11-17T05:57:35.692037+0000 mgr.smithi006.odexej (mgr.14176) 1104 : audit [DBG] from='client.15046 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:37.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:36 smithi156 bash[29655]: cluster 2023-11-17T05:57:35.361522+0000 mgr.smithi006.odexej (mgr.14176) 1103 : cluster [DBG] pgmap v768: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:37.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:36 smithi156 bash[29655]: audit 2023-11-17T05:57:35.692037+0000 mgr.smithi006.odexej (mgr.14176) 1104 : audit [DBG] from='client.15046 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:39.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:38 smithi006 bash[26022]: cluster 2023-11-17T05:57:37.362589+0000 mgr.smithi006.odexej (mgr.14176) 1105 : cluster [DBG] pgmap v769: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:39.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:38 smithi156 bash[29655]: cluster 2023-11-17T05:57:37.362589+0000 mgr.smithi006.odexej (mgr.14176) 1105 : cluster [DBG] pgmap v769: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:40.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:40.501 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:41.126 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:41.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:40 smithi006 bash[26022]: cluster 2023-11-17T05:57:39.364053+0000 mgr.smithi006.odexej (mgr.14176) 1106 : cluster [DBG] pgmap v770: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:41.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:40 smithi156 bash[29655]: cluster 2023-11-17T05:57:39.364053+0000 mgr.smithi006.odexej (mgr.14176) 1106 : cluster [DBG] pgmap v770: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:42.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:42.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:41 smithi006 bash[26022]: audit 2023-11-17T05:57:40.490832+0000 mgr.smithi006.odexej (mgr.14176) 1107 : audit [DBG] from='client.15050 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:42.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:41 smithi156 bash[29655]: audit 2023-11-17T05:57:40.490832+0000 mgr.smithi006.odexej (mgr.14176) 1107 : audit [DBG] from='client.15050 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:43.316 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:42 smithi006 bash[26022]: cluster 2023-11-17T05:57:41.365195+0000 mgr.smithi006.odexej (mgr.14176) 1108 : cluster [DBG] pgmap v771: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:43.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:42 smithi156 bash[29655]: cluster 2023-11-17T05:57:41.365195+0000 mgr.smithi006.odexej (mgr.14176) 1108 : cluster [DBG] pgmap v771: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:43 smithi006 bash[26022]: audit 2023-11-17T05:57:43.332211+0000 mon.smithi006 (mon.0) 935 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:43 smithi006 bash[26022]: audit 2023-11-17T05:57:43.642727+0000 mon.smithi006 (mon.0) 936 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:44 smithi006 bash[26022]: audit 2023-11-17T05:57:43.643907+0000 mon.smithi006 (mon.0) 937 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:44 smithi006 bash[26022]: audit 2023-11-17T05:57:43.650551+0000 mon.smithi006 (mon.0) 938 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:44 smithi006 bash[26022]: audit 2023-11-17T05:57:43.658540+0000 mon.smithi006 (mon.0) 939 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:57:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:44 smithi006 bash[26022]: audit 2023-11-17T05:57:43.672308+0000 mon.smithi006 (mon.0) 940 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:44.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.332211+0000 mon.smithi006 (mon.0) 935 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:57:44.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.642727+0000 mon.smithi006 (mon.0) 936 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:57:44.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.643907+0000 mon.smithi006 (mon.0) 937 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:57:44.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.650551+0000 mon.smithi006 (mon.0) 938 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:44.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.658540+0000 mon.smithi006 (mon.0) 939 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:57:44.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:44 smithi156 bash[29655]: audit 2023-11-17T05:57:43.672308+0000 mon.smithi006 (mon.0) 940 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:45.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:45 smithi006 bash[26022]: cluster 2023-11-17T05:57:43.366543+0000 mgr.smithi006.odexej (mgr.14176) 1109 : cluster [DBG] pgmap v772: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:45.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:45 smithi006 bash[26022]: cluster 2023-11-17T05:57:43.651330+0000 mgr.smithi006.odexej (mgr.14176) 1110 : cluster [DBG] pgmap v773: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:45.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:45 smithi006 bash[26022]: cephadm 2023-11-17T05:57:43.677117+0000 mgr.smithi006.odexej (mgr.14176) 1111 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ywhajb on smithi156 2023-11-17T05:57:45.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:45 smithi006 bash[26022]: cluster 2023-11-17T05:57:43.990525+0000 mon.smithi006 (mon.0) 941 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:57:45.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:45 smithi006 bash[26022]: cluster 2023-11-17T05:57:43.990591+0000 mon.smithi006 (mon.0) 942 : cluster [INF] Cluster is now healthy 2023-11-17T05:57:45.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:45 smithi156 bash[29655]: cluster 2023-11-17T05:57:43.366543+0000 mgr.smithi006.odexej (mgr.14176) 1109 : cluster [DBG] pgmap v772: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:57:45.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:45 smithi156 bash[29655]: cluster 2023-11-17T05:57:43.651330+0000 mgr.smithi006.odexej (mgr.14176) 1110 : cluster [DBG] pgmap v773: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:45.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:45 smithi156 bash[29655]: cephadm 2023-11-17T05:57:43.677117+0000 mgr.smithi006.odexej (mgr.14176) 1111 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ywhajb on smithi156 2023-11-17T05:57:45.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:45 smithi156 bash[29655]: cluster 2023-11-17T05:57:43.990525+0000 mon.smithi006 (mon.0) 941 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:57:45.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:45 smithi156 bash[29655]: cluster 2023-11-17T05:57:43.990591+0000 mon.smithi006 (mon.0) 942 : cluster [INF] Cluster is now healthy 2023-11-17T05:57:45.514 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:45.515 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:55:35.359704Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.qclyza on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:39.592012Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.hzgjze on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-hzgjze\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.hzgjze\nDeploy daemon haproxy.nfs.foo.smithi156.hzgjze ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.320789Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ditjtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ditjtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ditjtc\nDeploy daemon haproxy.nfs.foo.smithi006.ditjtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:56:43.325132Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.lnclvt on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:46.115 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:47.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:48.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:47 smithi006 bash[26022]: audit 2023-11-17T05:57:45.503465+0000 mgr.smithi006.odexej (mgr.14176) 1112 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:48.189 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:47 smithi006 bash[26022]: cluster 2023-11-17T05:57:45.652583+0000 mgr.smithi006.odexej (mgr.14176) 1113 : cluster [DBG] pgmap v774: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:48.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:47 smithi156 bash[29655]: audit 2023-11-17T05:57:45.503465+0000 mgr.smithi006.odexej (mgr.14176) 1112 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:48.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:47 smithi156 bash[29655]: cluster 2023-11-17T05:57:45.652583+0000 mgr.smithi006.odexej (mgr.14176) 1113 : cluster [DBG] pgmap v774: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:48.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:48 smithi006 bash[26022]: audit 2023-11-17T05:57:47.438058+0000 mon.smithi006 (mon.0) 943 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ywhajb"}]: dispatch 2023-11-17T05:57:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:48 smithi156 bash[29655]: audit 2023-11-17T05:57:47.438058+0000 mon.smithi006 (mon.0) 943 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ywhajb"}]: dispatch 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: cephadm 2023-11-17T05:57:47.436733+0000 mgr.smithi006.odexej (mgr.14176) 1114 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.385 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.386 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: cephadm 2023-11-17T05:57:47.437360+0000 mgr.smithi006.odexej (mgr.14176) 1115 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: cephadm 2023-11-17T05:57:47.440060+0000 mgr.smithi006.odexej (mgr.14176) 1116 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:49.387 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: cephadm 2023-11-17T05:57:47.444961+0000 mgr.smithi006.odexej (mgr.14176) 1117 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.gbdrtc on smithi006 2023-11-17T05:57:49.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:49 smithi006 bash[26022]: cluster 2023-11-17T05:57:47.653206+0000 mgr.smithi006.odexej (mgr.14176) 1118 : cluster [DBG] pgmap v775: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:49.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: cephadm 2023-11-17T05:57:47.436733+0000 mgr.smithi006.odexej (mgr.14176) 1114 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: cephadm 2023-11-17T05:57:47.437360+0000 mgr.smithi006.odexej (mgr.14176) 1115 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: cephadm 2023-11-17T05:57:47.440060+0000 mgr.smithi006.odexej (mgr.14176) 1116 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb 2023-11-17T05:57:49.448 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ywhajb ... 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: cephadm 2023-11-17T05:57:47.444961+0000 mgr.smithi006.odexej (mgr.14176) 1117 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.gbdrtc on smithi006 2023-11-17T05:57:49.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:49 smithi156 bash[29655]: cluster 2023-11-17T05:57:47.653206+0000 mgr.smithi006.odexej (mgr.14176) 1118 : cluster [DBG] pgmap v775: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:51.123 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:50 smithi006 bash[26022]: cluster 2023-11-17T05:57:49.654326+0000 mgr.smithi006.odexej (mgr.14176) 1119 : cluster [DBG] pgmap v776: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:51.123 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:50 smithi006 bash[26022]: audit 2023-11-17T05:57:49.729528+0000 mon.smithi006 (mon.0) 944 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:51.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:50 smithi156 bash[29655]: cluster 2023-11-17T05:57:49.654326+0000 mgr.smithi006.odexej (mgr.14176) 1119 : cluster [DBG] pgmap v776: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:57:51.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:50 smithi156 bash[29655]: audit 2023-11-17T05:57:49.729528+0000 mon.smithi006 (mon.0) 944 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:51.264 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:51.264 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:51.895 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:52.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:51 smithi006 bash[26022]: audit 2023-11-17T05:57:51.177304+0000 mon.smithi006 (mon.0) 945 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.gbdrtc"}]: dispatch 2023-11-17T05:57:52.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:51 smithi156 bash[29655]: audit 2023-11-17T05:57:51.177304+0000 mon.smithi006 (mon.0) 945 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.gbdrtc"}]: dispatch 2023-11-17T05:57:52.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:53.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cephadm 2023-11-17T05:57:51.176274+0000 mgr.smithi006.odexej (mgr.14176) 1120 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:53.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:53.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:53.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.061 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:54.061 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.061 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:54.061 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cephadm 2023-11-17T05:57:51.176764+0000 mgr.smithi006.odexej (mgr.14176) 1121 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cephadm 2023-11-17T05:57:51.178616+0000 mgr.smithi006.odexej (mgr.14176) 1122 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:54.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cephadm 2023-11-17T05:57:51.182860+0000 mgr.smithi006.odexej (mgr.14176) 1123 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cephadm 2023-11-17T05:57:51.184963+0000 mgr.smithi006.odexej (mgr.14176) 1124 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cluster 2023-11-17T05:57:51.185826+0000 mgr.smithi006.odexej (mgr.14176) 1125 : cluster [DBG] pgmap v777: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: audit 2023-11-17T05:57:51.254218+0000 mgr.smithi006.odexej (mgr.14176) 1126 : audit [DBG] from='client.15058 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:54.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:52 smithi006 bash[26022]: cluster 2023-11-17T05:57:51.729760+0000 mon.smithi006 (mon.0) 946 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cephadm 2023-11-17T05:57:51.176274+0000 mgr.smithi006.odexej (mgr.14176) 1120 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.064 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:54.065 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:57:54.454 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:54.455 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:54.456 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.498 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cephadm 2023-11-17T05:57:51.176764+0000 mgr.smithi006.odexej (mgr.14176) 1121 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.498 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cephadm 2023-11-17T05:57:51.178616+0000 mgr.smithi006.odexej (mgr.14176) 1122 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.498 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.498 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gbdrtc ... 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cephadm 2023-11-17T05:57:51.182860+0000 mgr.smithi006.odexej (mgr.14176) 1123 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cephadm 2023-11-17T05:57:51.184963+0000 mgr.smithi006.odexej (mgr.14176) 1124 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:57:54.499 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cluster 2023-11-17T05:57:51.185826+0000 mgr.smithi006.odexej (mgr.14176) 1125 : cluster [DBG] pgmap v777: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:57:54.500 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: audit 2023-11-17T05:57:51.254218+0000 mgr.smithi006.odexej (mgr.14176) 1126 : audit [DBG] from='client.15058 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:54.500 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:52 smithi156 bash[29655]: cluster 2023-11-17T05:57:51.729760+0000 mon.smithi006 (mon.0) 946 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:57:55.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:54 smithi006 bash[26022]: cluster 2023-11-17T05:57:53.186885+0000 mgr.smithi006.odexej (mgr.14176) 1127 : cluster [DBG] pgmap v778: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:57:55.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:54 smithi156 bash[29655]: cluster 2023-11-17T05:57:53.186885+0000 mgr.smithi006.odexej (mgr.14176) 1127 : cluster [DBG] pgmap v778: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:57:56.083 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:57:56.083 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:57:56.658 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:57:56.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:56 smithi006 bash[26022]: cluster 2023-11-17T05:57:55.188371+0000 mgr.smithi006.odexej (mgr.14176) 1128 : cluster [DBG] pgmap v779: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:57:56.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:56 smithi006 bash[26022]: audit 2023-11-17T05:57:55.828001+0000 mon.smithi006 (mon.0) 947 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:56.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:56 smithi006 bash[26022]: audit 2023-11-17T05:57:56.065652+0000 mgr.smithi006.odexej (mgr.14176) 1129 : audit [DBG] from='client.15062 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:57.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:56 smithi156 bash[29655]: cluster 2023-11-17T05:57:55.188371+0000 mgr.smithi006.odexej (mgr.14176) 1128 : cluster [DBG] pgmap v779: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:57:57.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:56 smithi156 bash[29655]: audit 2023-11-17T05:57:55.828001+0000 mon.smithi006 (mon.0) 947 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:57:57.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:56 smithi156 bash[29655]: audit 2023-11-17T05:57:56.065652+0000 mgr.smithi006.odexej (mgr.14176) 1129 : audit [DBG] from='client.15062 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:57:57.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:57:58.970 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:57:58 smithi006 bash[26022]: cluster 2023-11-17T05:57:57.189407+0000 mgr.smithi006.odexej (mgr.14176) 1130 : cluster [DBG] pgmap v780: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:57:59.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:57:58 smithi156 bash[29655]: cluster 2023-11-17T05:57:57.189407+0000 mgr.smithi006.odexej (mgr.14176) 1130 : cluster [DBG] pgmap v780: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:58:00.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:00.667 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:01.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:00 smithi006 bash[26022]: cluster 2023-11-17T05:57:59.190580+0000 mgr.smithi006.odexej (mgr.14176) 1131 : cluster [DBG] pgmap v781: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:58:01.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:00 smithi156 bash[29655]: cluster 2023-11-17T05:57:59.190580+0000 mgr.smithi006.odexej (mgr.14176) 1131 : cluster [DBG] pgmap v781: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:58:01.245 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:02.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:01 smithi006 bash[26022]: audit 2023-11-17T05:58:00.649191+0000 mgr.smithi006.odexej (mgr.14176) 1132 : audit [DBG] from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:02.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:01 smithi156 bash[29655]: audit 2023-11-17T05:58:00.649191+0000 mgr.smithi006.odexej (mgr.14176) 1132 : audit [DBG] from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:02.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:03.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:02 smithi006 bash[26022]: cluster 2023-11-17T05:58:01.192045+0000 mgr.smithi006.odexej (mgr.14176) 1133 : cluster [DBG] pgmap v782: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:58:03.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:02 smithi156 bash[29655]: cluster 2023-11-17T05:58:01.192045+0000 mgr.smithi006.odexej (mgr.14176) 1133 : cluster [DBG] pgmap v782: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:58:05.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:04 smithi006 bash[26022]: cluster 2023-11-17T05:58:03.193202+0000 mgr.smithi006.odexej (mgr.14176) 1134 : cluster [DBG] pgmap v783: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:05.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:04 smithi006 bash[26022]: audit 2023-11-17T05:58:04.728839+0000 mon.smithi006 (mon.0) 948 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:58:05.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:04 smithi156 bash[29655]: cluster 2023-11-17T05:58:03.193202+0000 mgr.smithi006.odexej (mgr.14176) 1134 : cluster [DBG] pgmap v783: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:05.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:04 smithi156 bash[29655]: audit 2023-11-17T05:58:04.728839+0000 mon.smithi006 (mon.0) 948 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:58:05.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:05.260 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:05.850 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:06.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:07.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:06 smithi006 bash[26022]: cluster 2023-11-17T05:58:05.194653+0000 mgr.smithi006.odexej (mgr.14176) 1135 : cluster [DBG] pgmap v784: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:07.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:06 smithi006 bash[26022]: audit 2023-11-17T05:58:05.245174+0000 mgr.smithi006.odexej (mgr.14176) 1136 : audit [DBG] from='client.15070 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:07.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:06 smithi156 bash[29655]: cluster 2023-11-17T05:58:05.194653+0000 mgr.smithi006.odexej (mgr.14176) 1135 : cluster [DBG] pgmap v784: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:07.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:06 smithi156 bash[29655]: audit 2023-11-17T05:58:05.245174+0000 mgr.smithi006.odexej (mgr.14176) 1136 : audit [DBG] from='client.15070 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:09.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:08 smithi006 bash[26022]: cluster 2023-11-17T05:58:07.195693+0000 mgr.smithi006.odexej (mgr.14176) 1137 : cluster [DBG] pgmap v785: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:09.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:08 smithi156 bash[29655]: cluster 2023-11-17T05:58:07.195693+0000 mgr.smithi006.odexej (mgr.14176) 1137 : cluster [DBG] pgmap v785: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:09.938 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:09.938 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:10.515 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:11.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:10 smithi006 bash[26022]: cluster 2023-11-17T05:58:09.196986+0000 mgr.smithi006.odexej (mgr.14176) 1138 : cluster [DBG] pgmap v786: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:11.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:10 smithi006 bash[26022]: audit 2023-11-17T05:58:09.922865+0000 mgr.smithi006.odexej (mgr.14176) 1139 : audit [DBG] from='client.15074 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:11.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:10 smithi156 bash[29655]: cluster 2023-11-17T05:58:09.196986+0000 mgr.smithi006.odexej (mgr.14176) 1138 : cluster [DBG] pgmap v786: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:11.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:10 smithi156 bash[29655]: audit 2023-11-17T05:58:09.922865+0000 mgr.smithi006.odexej (mgr.14176) 1139 : audit [DBG] from='client.15074 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:11.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:12.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:12 smithi006 bash[26022]: cluster 2023-11-17T05:58:11.198419+0000 mgr.smithi006.odexej (mgr.14176) 1140 : cluster [DBG] pgmap v787: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:12 smithi156 bash[29655]: cluster 2023-11-17T05:58:11.198419+0000 mgr.smithi006.odexej (mgr.14176) 1140 : cluster [DBG] pgmap v787: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:14.657 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:14.657 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:15.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:14 smithi006 bash[26022]: cluster 2023-11-17T05:58:13.199587+0000 mgr.smithi006.odexej (mgr.14176) 1141 : cluster [DBG] pgmap v788: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:15.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:14 smithi156 bash[29655]: cluster 2023-11-17T05:58:13.199587+0000 mgr.smithi006.odexej (mgr.14176) 1141 : cluster [DBG] pgmap v788: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:15.225 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:16.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:15 smithi006 bash[26022]: audit 2023-11-17T05:58:14.641794+0000 mgr.smithi006.odexej (mgr.14176) 1142 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:16.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:15 smithi156 bash[29655]: audit 2023-11-17T05:58:14.641794+0000 mgr.smithi006.odexej (mgr.14176) 1142 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:16.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:17.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:16 smithi006 bash[26022]: cluster 2023-11-17T05:58:15.201054+0000 mgr.smithi006.odexej (mgr.14176) 1143 : cluster [DBG] pgmap v789: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:17.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:16 smithi156 bash[29655]: cluster 2023-11-17T05:58:15.201054+0000 mgr.smithi006.odexej (mgr.14176) 1143 : cluster [DBG] pgmap v789: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:19.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:18 smithi006 bash[26022]: cluster 2023-11-17T05:58:17.202056+0000 mgr.smithi006.odexej (mgr.14176) 1144 : cluster [DBG] pgmap v790: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:19.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:18 smithi156 bash[29655]: cluster 2023-11-17T05:58:17.202056+0000 mgr.smithi006.odexej (mgr.14176) 1144 : cluster [DBG] pgmap v790: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:19.350 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:19.350 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:19.938 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:20.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:21.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:20 smithi156 bash[29655]: cluster 2023-11-17T05:58:19.202877+0000 mgr.smithi006.odexej (mgr.14176) 1145 : cluster [DBG] pgmap v791: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:21.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:20 smithi156 bash[29655]: audit 2023-11-17T05:58:19.340542+0000 mgr.smithi006.odexej (mgr.14176) 1146 : audit [DBG] from='client.15082 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:20 smithi006 bash[26022]: cluster 2023-11-17T05:58:19.202877+0000 mgr.smithi006.odexej (mgr.14176) 1145 : cluster [DBG] pgmap v791: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:21.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:20 smithi006 bash[26022]: audit 2023-11-17T05:58:19.340542+0000 mgr.smithi006.odexej (mgr.14176) 1146 : audit [DBG] from='client.15082 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:23.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:22 smithi156 bash[29655]: cluster 2023-11-17T05:58:21.204219+0000 mgr.smithi006.odexej (mgr.14176) 1147 : cluster [DBG] pgmap v792: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:23.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:22 smithi006 bash[26022]: cluster 2023-11-17T05:58:21.204219+0000 mgr.smithi006.odexej (mgr.14176) 1147 : cluster [DBG] pgmap v792: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:24.024 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:24.025 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:24.644 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:25.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:24 smithi156 bash[29655]: cluster 2023-11-17T05:58:23.205296+0000 mgr.smithi006.odexej (mgr.14176) 1148 : cluster [DBG] pgmap v793: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:25.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:24 smithi156 bash[29655]: audit 2023-11-17T05:58:24.006270+0000 mgr.smithi006.odexej (mgr.14176) 1149 : audit [DBG] from='client.15086 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:24 smithi006 bash[26022]: cluster 2023-11-17T05:58:23.205296+0000 mgr.smithi006.odexej (mgr.14176) 1148 : cluster [DBG] pgmap v793: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:24 smithi006 bash[26022]: audit 2023-11-17T05:58:24.006270+0000 mgr.smithi006.odexej (mgr.14176) 1149 : audit [DBG] from='client.15086 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:25.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:27.040 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:26 smithi006 bash[26022]: cluster 2023-11-17T05:58:25.206728+0000 mgr.smithi006.odexej (mgr.14176) 1150 : cluster [DBG] pgmap v794: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:27.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:26 smithi156 bash[29655]: cluster 2023-11-17T05:58:25.206728+0000 mgr.smithi006.odexej (mgr.14176) 1150 : cluster [DBG] pgmap v794: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:28.742 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:28.742 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:29.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:28 smithi006 bash[26022]: cluster 2023-11-17T05:58:27.207914+0000 mgr.smithi006.odexej (mgr.14176) 1151 : cluster [DBG] pgmap v795: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:29.376 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:28 smithi156 bash[29655]: cluster 2023-11-17T05:58:27.207914+0000 mgr.smithi006.odexej (mgr.14176) 1151 : cluster [DBG] pgmap v795: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:30.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:30.389 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:29 smithi006 bash[26022]: audit 2023-11-17T05:58:28.730405+0000 mgr.smithi006.odexej (mgr.14176) 1152 : audit [DBG] from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:30.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:29 smithi156 bash[29655]: audit 2023-11-17T05:58:28.730405+0000 mgr.smithi006.odexej (mgr.14176) 1152 : audit [DBG] from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:31.365 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:30 smithi006 bash[26022]: cluster 2023-11-17T05:58:29.209116+0000 mgr.smithi006.odexej (mgr.14176) 1153 : cluster [DBG] pgmap v796: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:31.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:30 smithi156 bash[29655]: cluster 2023-11-17T05:58:29.209116+0000 mgr.smithi006.odexej (mgr.14176) 1153 : cluster [DBG] pgmap v796: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:33.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:33.380 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:33.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:32 smithi006 bash[26022]: cluster 2023-11-17T05:58:31.210471+0000 mgr.smithi006.odexej (mgr.14176) 1154 : cluster [DBG] pgmap v797: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:33.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:32 smithi156 bash[29655]: cluster 2023-11-17T05:58:31.210471+0000 mgr.smithi006.odexej (mgr.14176) 1154 : cluster [DBG] pgmap v797: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:33.963 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:34.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:35.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:34 smithi006 bash[26022]: cluster 2023-11-17T05:58:33.211488+0000 mgr.smithi006.odexej (mgr.14176) 1155 : cluster [DBG] pgmap v798: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:35.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:34 smithi006 bash[26022]: audit 2023-11-17T05:58:33.368012+0000 mgr.smithi006.odexej (mgr.14176) 1156 : audit [DBG] from='client.15094 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:35.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:34 smithi156 bash[29655]: cluster 2023-11-17T05:58:33.211488+0000 mgr.smithi006.odexej (mgr.14176) 1155 : cluster [DBG] pgmap v798: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:35.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:34 smithi156 bash[29655]: audit 2023-11-17T05:58:33.368012+0000 mgr.smithi006.odexej (mgr.14176) 1156 : audit [DBG] from='client.15094 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:37.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:36 smithi006 bash[26022]: cluster 2023-11-17T05:58:35.212912+0000 mgr.smithi006.odexej (mgr.14176) 1157 : cluster [DBG] pgmap v799: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:37.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:36 smithi156 bash[29655]: cluster 2023-11-17T05:58:35.212912+0000 mgr.smithi006.odexej (mgr.14176) 1157 : cluster [DBG] pgmap v799: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:38.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:38.123 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:38.714 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:39.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:38 smithi006 bash[26022]: cluster 2023-11-17T05:58:37.214003+0000 mgr.smithi006.odexej (mgr.14176) 1158 : cluster [DBG] pgmap v800: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:39.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:38 smithi006 bash[26022]: audit 2023-11-17T05:58:38.113268+0000 mgr.smithi006.odexej (mgr.14176) 1159 : audit [DBG] from='client.15098 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:39.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:38 smithi156 bash[29655]: cluster 2023-11-17T05:58:37.214003+0000 mgr.smithi006.odexej (mgr.14176) 1158 : cluster [DBG] pgmap v800: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:39.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:38 smithi156 bash[29655]: audit 2023-11-17T05:58:38.113268+0000 mgr.smithi006.odexej (mgr.14176) 1159 : audit [DBG] from='client.15098 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:39.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:41.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:40 smithi006 bash[26022]: cluster 2023-11-17T05:58:39.215723+0000 mgr.smithi006.odexej (mgr.14176) 1160 : cluster [DBG] pgmap v801: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:41.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:40 smithi156 bash[29655]: cluster 2023-11-17T05:58:39.215723+0000 mgr.smithi006.odexej (mgr.14176) 1160 : cluster [DBG] pgmap v801: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:42.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:42.637 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:43.200 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:43.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:42 smithi006 bash[26022]: cluster 2023-11-17T05:58:41.217250+0000 mgr.smithi006.odexej (mgr.14176) 1161 : cluster [DBG] pgmap v802: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:43.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:42 smithi156 bash[29655]: cluster 2023-11-17T05:58:41.217250+0000 mgr.smithi006.odexej (mgr.14176) 1161 : cluster [DBG] pgmap v802: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:44.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:43 smithi006 bash[26022]: audit 2023-11-17T05:58:42.621628+0000 mgr.smithi006.odexej (mgr.14176) 1162 : audit [DBG] from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:44.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:43 smithi156 bash[29655]: audit 2023-11-17T05:58:42.621628+0000 mgr.smithi006.odexej (mgr.14176) 1162 : audit [DBG] from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:45.271 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:44 smithi006 bash[26022]: cluster 2023-11-17T05:58:43.218189+0000 mgr.smithi006.odexej (mgr.14176) 1163 : cluster [DBG] pgmap v803: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:45.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:44 smithi156 bash[29655]: cluster 2023-11-17T05:58:43.218189+0000 mgr.smithi006.odexej (mgr.14176) 1163 : cluster [DBG] pgmap v803: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:58:47.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:58:47.141 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:58:47.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:47 smithi006 bash[26022]: cluster 2023-11-17T05:58:45.219402+0000 mgr.smithi006.odexej (mgr.14176) 1164 : cluster [DBG] pgmap v804: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:47.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:47 smithi156 bash[29655]: cluster 2023-11-17T05:58:45.219402+0000 mgr.smithi006.odexej (mgr.14176) 1164 : cluster [DBG] pgmap v804: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:47.824 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:58:48.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:48 smithi006 bash[26022]: audit 2023-11-17T05:58:47.126201+0000 mgr.smithi006.odexej (mgr.14176) 1165 : audit [DBG] from='client.15106 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:48 smithi156 bash[29655]: audit 2023-11-17T05:58:47.126201+0000 mgr.smithi006.odexej (mgr.14176) 1165 : audit [DBG] from='client.15106 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:58:48.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:58:49.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:49 smithi006 bash[26022]: cluster 2023-11-17T05:58:47.221213+0000 mgr.smithi006.odexej (mgr.14176) 1166 : cluster [DBG] pgmap v805: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:49.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:49 smithi156 bash[29655]: cluster 2023-11-17T05:58:47.221213+0000 mgr.smithi006.odexej (mgr.14176) 1166 : cluster [DBG] pgmap v805: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:58:51.343 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:51 smithi006 bash[26022]: cluster 2023-11-17T05:58:49.222391+0000 mgr.smithi006.odexej (mgr.14176) 1167 : cluster [DBG] pgmap v806: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:09.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:09.799 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:56:43.328279Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.akrgva on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:47.439728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ywhajb on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ywhajb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ywhajb\nDeploy daemon haproxy.nfs.foo.smithi156.ywhajb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.178373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gbdrtc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gbdrtc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gbdrtc\nDeploy daemon haproxy.nfs.foo.smithi006.gbdrtc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:57:51.182647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.dbsnlm on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.188830+0000 mon.smithi006 (mon.0) 949 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.502055+0000 mon.smithi006 (mon.0) 950 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.503348+0000 mon.smithi006 (mon.0) 951 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.511279+0000 mon.smithi006 (mon.0) 952 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.517698+0000 mon.smithi006 (mon.0) 953 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:59:09.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:52 smithi006 bash[26022]: audit 2023-11-17T05:58:51.528863+0000 mon.smithi006 (mon.0) 954 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:09.802 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:09.981 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:51 smithi156 bash[29655]: cluster 2023-11-17T05:58:49.222391+0000 mgr.smithi006.odexej (mgr.14176) 1167 : cluster [DBG] pgmap v806: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:09.981 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.188830+0000 mon.smithi006 (mon.0) 949 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.502055+0000 mon.smithi006 (mon.0) 950 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.503348+0000 mon.smithi006 (mon.0) 951 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.511279+0000 mon.smithi006 (mon.0) 952 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.517698+0000 mon.smithi006 (mon.0) 953 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:52 smithi156 bash[29655]: audit 2023-11-17T05:58:51.528863+0000 mon.smithi006 (mon.0) 954 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:10.007 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: cluster 2023-11-17T05:58:51.223389+0000 mgr.smithi006.odexej (mgr.14176) 1168 : cluster [DBG] pgmap v807: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: cluster 2023-11-17T05:58:51.512066+0000 mgr.smithi006.odexej (mgr.14176) 1169 : cluster [DBG] pgmap v808: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: cephadm 2023-11-17T05:58:51.532321+0000 mgr.smithi006.odexej (mgr.14176) 1170 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.egadkh on smithi156 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: audit 2023-11-17T05:58:51.735008+0000 mgr.smithi006.odexej (mgr.14176) 1171 : audit [DBG] from='client.15110 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: cluster 2023-11-17T05:58:52.021531+0000 mon.smithi006 (mon.0) 955 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:53 smithi156 bash[29655]: cluster 2023-11-17T05:58:52.021609+0000 mon.smithi006 (mon.0) 956 : cluster [INF] Cluster is now healthy 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:55 smithi156 bash[29655]: cluster 2023-11-17T05:58:53.516038+0000 mgr.smithi006.odexej (mgr.14176) 1172 : cluster [DBG] pgmap v809: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:56 smithi156 bash[29655]: audit 2023-11-17T05:58:55.294292+0000 mon.smithi006 (mon.0) 957 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.egadkh"}]: dispatch 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: cephadm 2023-11-17T05:58:55.293013+0000 mgr.smithi006.odexej (mgr.14176) 1173 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.008 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:59:10.009 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.010 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.010 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.010 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.010 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.011 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.011 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.130 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.130 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: cephadm 2023-11-17T05:58:55.293512+0000 mgr.smithi006.odexej (mgr.14176) 1174 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: cephadm 2023-11-17T05:58:55.296194+0000 mgr.smithi006.odexej (mgr.14176) 1175 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.131 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: cephadm 2023-11-17T05:58:55.301207+0000 mgr.smithi006.odexej (mgr.14176) 1176 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.xmrwce on smithi006 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:57 smithi156 bash[29655]: cluster 2023-11-17T05:58:55.516972+0000 mgr.smithi006.odexej (mgr.14176) 1177 : cluster [DBG] pgmap v810: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:59 smithi156 bash[29655]: cluster 2023-11-17T05:58:57.517738+0000 mgr.smithi006.odexej (mgr.14176) 1178 : cluster [DBG] pgmap v811: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:58:59 smithi156 bash[29655]: audit 2023-11-17T05:58:59.026923+0000 mon.smithi006 (mon.0) 958 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.xmrwce"}]: dispatch 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cephadm 2023-11-17T05:58:59.025809+0000 mgr.smithi006.odexej (mgr.14176) 1179 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.132 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T05:59:10.133 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:59:10.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T05:59:10.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:59:10.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T05:59:10.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.134 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cephadm 2023-11-17T05:58:59.026321+0000 mgr.smithi006.odexej (mgr.14176) 1180 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.209 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cephadm 2023-11-17T05:58:59.028622+0000 mgr.smithi006.odexej (mgr.14176) 1181 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.210 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cephadm 2023-11-17T05:58:59.033269+0000 mgr.smithi006.odexej (mgr.14176) 1182 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cephadm 2023-11-17T05:58:59.036544+0000 mgr.smithi006.odexej (mgr.14176) 1183 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cluster 2023-11-17T05:58:59.037757+0000 mgr.smithi006.odexej (mgr.14176) 1184 : cluster [DBG] pgmap v812: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:00 smithi156 bash[29655]: cluster 2023-11-17T05:58:59.057249+0000 mon.smithi006 (mon.0) 959 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:01 smithi156 bash[29655]: audit 2023-11-17T05:59:00.882329+0000 mon.smithi006 (mon.0) 960 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:01 smithi156 bash[29655]: cluster 2023-11-17T05:59:01.039164+0000 mgr.smithi006.odexej (mgr.14176) 1185 : cluster [DBG] pgmap v813: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:04 smithi156 bash[29655]: cluster 2023-11-17T05:59:03.040236+0000 mgr.smithi006.odexej (mgr.14176) 1186 : cluster [DBG] pgmap v814: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:06 smithi156 bash[29655]: cluster 2023-11-17T05:59:05.041632+0000 mgr.smithi006.odexej (mgr.14176) 1187 : cluster [DBG] pgmap v815: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.211 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:07 smithi156 bash[29655]: cluster 2023-11-17T05:59:07.042633+0000 mgr.smithi006.odexej (mgr.14176) 1188 : cluster [DBG] pgmap v816: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: cluster 2023-11-17T05:58:51.223389+0000 mgr.smithi006.odexej (mgr.14176) 1168 : cluster [DBG] pgmap v807: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:10.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: cluster 2023-11-17T05:58:51.512066+0000 mgr.smithi006.odexej (mgr.14176) 1169 : cluster [DBG] pgmap v808: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: cephadm 2023-11-17T05:58:51.532321+0000 mgr.smithi006.odexej (mgr.14176) 1170 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.egadkh on smithi156 2023-11-17T05:59:10.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: audit 2023-11-17T05:58:51.735008+0000 mgr.smithi006.odexej (mgr.14176) 1171 : audit [DBG] from='client.15110 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: cluster 2023-11-17T05:58:52.021531+0000 mon.smithi006 (mon.0) 955 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:53 smithi006 bash[26022]: cluster 2023-11-17T05:58:52.021609+0000 mon.smithi006 (mon.0) 956 : cluster [INF] Cluster is now healthy 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:55 smithi006 bash[26022]: cluster 2023-11-17T05:58:53.516038+0000 mgr.smithi006.odexej (mgr.14176) 1172 : cluster [DBG] pgmap v809: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:56 smithi006 bash[26022]: audit 2023-11-17T05:58:55.294292+0000 mon.smithi006 (mon.0) 957 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.egadkh"}]: dispatch 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: cephadm 2023-11-17T05:58:55.293013+0000 mgr.smithi006.odexej (mgr.14176) 1173 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: cephadm 2023-11-17T05:58:55.293512+0000 mgr.smithi006.odexej (mgr.14176) 1174 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: cephadm 2023-11-17T05:58:55.296194+0000 mgr.smithi006.odexej (mgr.14176) 1175 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.265 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.265 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh 2023-11-17T05:59:10.265 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.egadkh ... 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: cephadm 2023-11-17T05:58:55.301207+0000 mgr.smithi006.odexej (mgr.14176) 1176 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.xmrwce on smithi006 2023-11-17T05:59:10.266 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:57 smithi006 bash[26022]: cluster 2023-11-17T05:58:55.516972+0000 mgr.smithi006.odexej (mgr.14176) 1177 : cluster [DBG] pgmap v810: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:59 smithi006 bash[26022]: cluster 2023-11-17T05:58:57.517738+0000 mgr.smithi006.odexej (mgr.14176) 1178 : cluster [DBG] pgmap v811: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:58:59 smithi006 bash[26022]: audit 2023-11-17T05:58:59.026923+0000 mon.smithi006 (mon.0) 958 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.xmrwce"}]: dispatch 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cephadm 2023-11-17T05:58:59.025809+0000 mgr.smithi006.odexej (mgr.14176) 1179 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.267 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.268 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.294 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.294 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.294 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.294 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.294 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cephadm 2023-11-17T05:58:59.026321+0000 mgr.smithi006.odexej (mgr.14176) 1180 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cephadm 2023-11-17T05:58:59.028622+0000 mgr.smithi006.odexej (mgr.14176) 1181 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.295 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xmrwce ... 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cephadm 2023-11-17T05:58:59.033269+0000 mgr.smithi006.odexej (mgr.14176) 1182 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cephadm 2023-11-17T05:58:59.036544+0000 mgr.smithi006.odexej (mgr.14176) 1183 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cluster 2023-11-17T05:58:59.037757+0000 mgr.smithi006.odexej (mgr.14176) 1184 : cluster [DBG] pgmap v812: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:59:10.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:00 smithi006 bash[26022]: cluster 2023-11-17T05:58:59.057249+0000 mon.smithi006 (mon.0) 959 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T05:59:10.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:01 smithi006 bash[26022]: audit 2023-11-17T05:59:00.882329+0000 mon.smithi006 (mon.0) 960 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T05:59:10.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:01 smithi006 bash[26022]: cluster 2023-11-17T05:59:01.039164+0000 mgr.smithi006.odexej (mgr.14176) 1185 : cluster [DBG] pgmap v813: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T05:59:10.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:04 smithi006 bash[26022]: cluster 2023-11-17T05:59:03.040236+0000 mgr.smithi006.odexej (mgr.14176) 1186 : cluster [DBG] pgmap v814: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T05:59:10.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:06 smithi006 bash[26022]: cluster 2023-11-17T05:59:05.041632+0000 mgr.smithi006.odexej (mgr.14176) 1187 : cluster [DBG] pgmap v815: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:07 smithi006 bash[26022]: cluster 2023-11-17T05:59:07.042633+0000 mgr.smithi006.odexej (mgr.14176) 1188 : cluster [DBG] pgmap v816: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:10 smithi006 bash[26022]: cluster 2023-11-17T05:59:09.043825+0000 mgr.smithi006.odexej (mgr.14176) 1189 : cluster [DBG] pgmap v817: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:10 smithi156 bash[29655]: cluster 2023-11-17T05:59:09.043825+0000 mgr.smithi006.odexej (mgr.14176) 1189 : cluster [DBG] pgmap v817: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T05:59:10.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:12.281 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:12 smithi006 bash[26022]: cluster 2023-11-17T05:59:11.045202+0000 mgr.smithi006.odexej (mgr.14176) 1190 : cluster [DBG] pgmap v818: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:12.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:12 smithi156 bash[29655]: cluster 2023-11-17T05:59:11.045202+0000 mgr.smithi006.odexej (mgr.14176) 1190 : cluster [DBG] pgmap v818: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:13.990 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:13.990 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:14.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:14 smithi006 bash[26022]: cluster 2023-11-17T05:59:13.046252+0000 mgr.smithi006.odexej (mgr.14176) 1191 : cluster [DBG] pgmap v819: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:14.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:14 smithi156 bash[29655]: cluster 2023-11-17T05:59:13.046252+0000 mgr.smithi006.odexej (mgr.14176) 1191 : cluster [DBG] pgmap v819: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:14.595 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:15.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:15 smithi006 bash[26022]: audit 2023-11-17T05:59:13.975700+0000 mgr.smithi006.odexej (mgr.14176) 1192 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:15.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:15 smithi156 bash[29655]: audit 2023-11-17T05:59:13.975700+0000 mgr.smithi006.odexej (mgr.14176) 1192 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:15.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:16.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:16 smithi006 bash[26022]: cluster 2023-11-17T05:59:15.047673+0000 mgr.smithi006.odexej (mgr.14176) 1193 : cluster [DBG] pgmap v820: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:16.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:16 smithi156 bash[29655]: cluster 2023-11-17T05:59:15.047673+0000 mgr.smithi006.odexej (mgr.14176) 1193 : cluster [DBG] pgmap v820: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:17.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:17 smithi156 bash[29655]: cluster 2023-11-17T05:59:17.048502+0000 mgr.smithi006.odexej (mgr.14176) 1194 : cluster [DBG] pgmap v821: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:18.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:17 smithi006 bash[26022]: cluster 2023-11-17T05:59:17.048502+0000 mgr.smithi006.odexej (mgr.14176) 1194 : cluster [DBG] pgmap v821: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:18.864 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:19.474 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:20.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:20 smithi006 bash[26022]: audit 2023-11-17T05:59:18.846821+0000 mgr.smithi006.odexej (mgr.14176) 1195 : audit [DBG] from='client.15118 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:20.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:20 smithi006 bash[26022]: cluster 2023-11-17T05:59:19.049680+0000 mgr.smithi006.odexej (mgr.14176) 1196 : cluster [DBG] pgmap v822: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:20.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:20 smithi156 bash[29655]: audit 2023-11-17T05:59:18.846821+0000 mgr.smithi006.odexej (mgr.14176) 1195 : audit [DBG] from='client.15118 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:20.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:20 smithi156 bash[29655]: cluster 2023-11-17T05:59:19.049680+0000 mgr.smithi006.odexej (mgr.14176) 1196 : cluster [DBG] pgmap v822: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:20.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:22.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:22 smithi006 bash[26022]: cluster 2023-11-17T05:59:21.051058+0000 mgr.smithi006.odexej (mgr.14176) 1197 : cluster [DBG] pgmap v823: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:22.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:22 smithi156 bash[29655]: cluster 2023-11-17T05:59:21.051058+0000 mgr.smithi006.odexej (mgr.14176) 1197 : cluster [DBG] pgmap v823: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:23.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:23.689 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:24.302 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:24.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:24 smithi006 bash[26022]: cluster 2023-11-17T05:59:23.052157+0000 mgr.smithi006.odexej (mgr.14176) 1198 : cluster [DBG] pgmap v824: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:24.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:24 smithi156 bash[29655]: cluster 2023-11-17T05:59:23.052157+0000 mgr.smithi006.odexej (mgr.14176) 1198 : cluster [DBG] pgmap v824: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:25.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:25 smithi006 bash[26022]: audit 2023-11-17T05:59:23.677596+0000 mgr.smithi006.odexej (mgr.14176) 1199 : audit [DBG] from='client.15122 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:25 smithi156 bash[29655]: audit 2023-11-17T05:59:23.677596+0000 mgr.smithi006.odexej (mgr.14176) 1199 : audit [DBG] from='client.15122 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:26.399 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:26 smithi006 bash[26022]: cluster 2023-11-17T05:59:25.053587+0000 mgr.smithi006.odexej (mgr.14176) 1200 : cluster [DBG] pgmap v825: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:26.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:26 smithi156 bash[29655]: cluster 2023-11-17T05:59:25.053587+0000 mgr.smithi006.odexej (mgr.14176) 1200 : cluster [DBG] pgmap v825: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:27.931 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:27 smithi006 bash[26022]: cluster 2023-11-17T05:59:27.054636+0000 mgr.smithi006.odexej (mgr.14176) 1201 : cluster [DBG] pgmap v826: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:27.933 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:27 smithi156 bash[29655]: cluster 2023-11-17T05:59:27.054636+0000 mgr.smithi006.odexej (mgr.14176) 1201 : cluster [DBG] pgmap v826: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:28.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:28.295 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:29.005 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:30.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:30.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:30 smithi006 bash[26022]: audit 2023-11-17T05:59:28.280802+0000 mgr.smithi006.odexej (mgr.14176) 1202 : audit [DBG] from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:30.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:30 smithi006 bash[26022]: cluster 2023-11-17T05:59:29.055891+0000 mgr.smithi006.odexej (mgr.14176) 1203 : cluster [DBG] pgmap v827: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:30 smithi156 bash[29655]: audit 2023-11-17T05:59:28.280802+0000 mgr.smithi006.odexej (mgr.14176) 1202 : audit [DBG] from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:30 smithi156 bash[29655]: cluster 2023-11-17T05:59:29.055891+0000 mgr.smithi006.odexej (mgr.14176) 1203 : cluster [DBG] pgmap v827: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:32.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:32 smithi006 bash[26022]: cluster 2023-11-17T05:59:31.057111+0000 mgr.smithi006.odexej (mgr.14176) 1204 : cluster [DBG] pgmap v828: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:32.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:32 smithi156 bash[29655]: cluster 2023-11-17T05:59:31.057111+0000 mgr.smithi006.odexej (mgr.14176) 1204 : cluster [DBG] pgmap v828: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:32.988 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:32.988 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:33.613 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:34.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:34 smithi006 bash[26022]: audit 2023-11-17T05:59:32.965897+0000 mgr.smithi006.odexej (mgr.14176) 1205 : audit [DBG] from='client.15130 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:34.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:34 smithi006 bash[26022]: cluster 2023-11-17T05:59:33.057981+0000 mgr.smithi006.odexej (mgr.14176) 1206 : cluster [DBG] pgmap v829: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:34.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:34 smithi156 bash[29655]: audit 2023-11-17T05:59:32.965897+0000 mgr.smithi006.odexej (mgr.14176) 1205 : audit [DBG] from='client.15130 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:34.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:34 smithi156 bash[29655]: cluster 2023-11-17T05:59:33.057981+0000 mgr.smithi006.odexej (mgr.14176) 1206 : cluster [DBG] pgmap v829: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:34.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:36.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:36 smithi006 bash[26022]: cluster 2023-11-17T05:59:35.059366+0000 mgr.smithi006.odexej (mgr.14176) 1207 : cluster [DBG] pgmap v830: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:36.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:36 smithi156 bash[29655]: cluster 2023-11-17T05:59:35.059366+0000 mgr.smithi006.odexej (mgr.14176) 1207 : cluster [DBG] pgmap v830: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:37.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:37.744 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:38.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:37 smithi006 bash[26022]: cluster 2023-11-17T05:59:37.060507+0000 mgr.smithi006.odexej (mgr.14176) 1208 : cluster [DBG] pgmap v831: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:38.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:37 smithi156 bash[29655]: cluster 2023-11-17T05:59:37.060507+0000 mgr.smithi006.odexej (mgr.14176) 1208 : cluster [DBG] pgmap v831: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:38.353 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:39.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:38 smithi006 bash[26022]: audit 2023-11-17T05:59:37.728606+0000 mgr.smithi006.odexej (mgr.14176) 1209 : audit [DBG] from='client.15134 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:39.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:38 smithi156 bash[29655]: audit 2023-11-17T05:59:37.728606+0000 mgr.smithi006.odexej (mgr.14176) 1209 : audit [DBG] from='client.15134 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:39.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:39 smithi006 bash[26022]: cluster 2023-11-17T05:59:39.061742+0000 mgr.smithi006.odexej (mgr.14176) 1210 : cluster [DBG] pgmap v832: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:39 smithi156 bash[29655]: cluster 2023-11-17T05:59:39.061742+0000 mgr.smithi006.odexej (mgr.14176) 1210 : cluster [DBG] pgmap v832: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:42.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:42 smithi006 bash[26022]: cluster 2023-11-17T05:59:41.063119+0000 mgr.smithi006.odexej (mgr.14176) 1211 : cluster [DBG] pgmap v833: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:42.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:42 smithi156 bash[29655]: cluster 2023-11-17T05:59:41.063119+0000 mgr.smithi006.odexej (mgr.14176) 1211 : cluster [DBG] pgmap v833: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:42.525 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:42.525 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:43.138 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:44.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:44 smithi006 bash[26022]: audit 2023-11-17T05:59:42.509870+0000 mgr.smithi006.odexej (mgr.14176) 1212 : audit [DBG] from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:44 smithi006 bash[26022]: cluster 2023-11-17T05:59:43.064286+0000 mgr.smithi006.odexej (mgr.14176) 1213 : cluster [DBG] pgmap v834: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:44.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:44 smithi156 bash[29655]: audit 2023-11-17T05:59:42.509870+0000 mgr.smithi006.odexej (mgr.14176) 1212 : audit [DBG] from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:44.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:44 smithi156 bash[29655]: cluster 2023-11-17T05:59:43.064286+0000 mgr.smithi006.odexej (mgr.14176) 1213 : cluster [DBG] pgmap v834: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:46.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:46 smithi006 bash[26022]: cluster 2023-11-17T05:59:45.065755+0000 mgr.smithi006.odexej (mgr.14176) 1214 : cluster [DBG] pgmap v835: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:46.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:46 smithi156 bash[29655]: cluster 2023-11-17T05:59:45.065755+0000 mgr.smithi006.odexej (mgr.14176) 1214 : cluster [DBG] pgmap v835: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:47.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:47.161 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:47.711 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:48.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:47 smithi006 bash[26022]: cluster 2023-11-17T05:59:47.066815+0000 mgr.smithi006.odexej (mgr.14176) 1215 : cluster [DBG] pgmap v836: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:48.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:47 smithi006 bash[26022]: audit 2023-11-17T05:59:47.152381+0000 mgr.smithi006.odexej (mgr.14176) 1216 : audit [DBG] from='client.15142 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:48.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:47 smithi156 bash[29655]: cluster 2023-11-17T05:59:47.066815+0000 mgr.smithi006.odexej (mgr.14176) 1215 : cluster [DBG] pgmap v836: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:48.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:47 smithi156 bash[29655]: audit 2023-11-17T05:59:47.152381+0000 mgr.smithi006.odexej (mgr.14176) 1216 : audit [DBG] from='client.15142 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:48.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:50.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:50 smithi006 bash[26022]: cluster 2023-11-17T05:59:49.068019+0000 mgr.smithi006.odexej (mgr.14176) 1217 : cluster [DBG] pgmap v837: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:57.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:50 smithi156 bash[29655]: cluster 2023-11-17T05:59:49.068019+0000 mgr.smithi006.odexej (mgr.14176) 1217 : cluster [DBG] pgmap v837: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:57.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:52 smithi156 bash[29655]: cluster 2023-11-17T05:59:51.069401+0000 mgr.smithi006.odexej (mgr.14176) 1218 : cluster [DBG] pgmap v838: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:57.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:53 smithi156 bash[29655]: audit 2023-11-17T05:59:51.702421+0000 mgr.smithi006.odexej (mgr.14176) 1219 : audit [DBG] from='client.15146 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:57.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:54 smithi156 bash[29655]: cluster 2023-11-17T05:59:53.070469+0000 mgr.smithi006.odexej (mgr.14176) 1220 : cluster [DBG] pgmap v839: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:57.282 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:56 smithi156 bash[29655]: cluster 2023-11-17T05:59:55.071903+0000 mgr.smithi006.odexej (mgr.14176) 1221 : cluster [DBG] pgmap v840: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:57.282 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T05:59:57.283 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T05:59:57.283 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T05:59:57.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:52 smithi006 bash[26022]: cluster 2023-11-17T05:59:51.069401+0000 mgr.smithi006.odexej (mgr.14176) 1218 : cluster [DBG] pgmap v838: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:57.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:53 smithi006 bash[26022]: audit 2023-11-17T05:59:51.702421+0000 mgr.smithi006.odexej (mgr.14176) 1219 : audit [DBG] from='client.15146 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:59:57.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:54 smithi006 bash[26022]: cluster 2023-11-17T05:59:53.070469+0000 mgr.smithi006.odexej (mgr.14176) 1220 : cluster [DBG] pgmap v839: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T05:59:57.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:56 smithi006 bash[26022]: cluster 2023-11-17T05:59:55.071903+0000 mgr.smithi006.odexej (mgr.14176) 1221 : cluster [DBG] pgmap v840: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:58.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:57 smithi006 bash[26022]: cluster 2023-11-17T05:59:57.072972+0000 mgr.smithi006.odexej (mgr.14176) 1222 : cluster [DBG] pgmap v841: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:58.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:57 smithi156 bash[29655]: cluster 2023-11-17T05:59:57.072972+0000 mgr.smithi006.odexej (mgr.14176) 1222 : cluster [DBG] pgmap v841: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T05:59:58.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T05:59:59.399 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 05:59:59 smithi006 bash[26022]: audit 2023-11-17T05:59:59.040346+0000 mon.smithi006 (mon.0) 961 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:59:59.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 05:59:59 smithi156 bash[29655]: audit 2023-11-17T05:59:59.040346+0000 mon.smithi006 (mon.0) 961 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: cluster 2023-11-17T05:59:59.074116+0000 mgr.smithi006.odexej (mgr.14176) 1223 : cluster [DBG] pgmap v842: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: audit 2023-11-17T05:59:59.356215+0000 mon.smithi006 (mon.0) 962 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: audit 2023-11-17T05:59:59.357552+0000 mon.smithi006 (mon.0) 963 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: audit 2023-11-17T05:59:59.367630+0000 mon.smithi006 (mon.0) 964 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: audit 2023-11-17T05:59:59.377267+0000 mon.smithi006 (mon.0) 965 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: audit 2023-11-17T05:59:59.389295+0000 mon.smithi006 (mon.0) 966 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:00.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:00 smithi006 bash[26022]: cluster 2023-11-17T06:00:00.000131+0000 mon.smithi006 (mon.0) 967 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T06:00:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: cluster 2023-11-17T05:59:59.074116+0000 mgr.smithi006.odexej (mgr.14176) 1223 : cluster [DBG] pgmap v842: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: audit 2023-11-17T05:59:59.356215+0000 mon.smithi006 (mon.0) 962 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:00:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: audit 2023-11-17T05:59:59.357552+0000 mon.smithi006 (mon.0) 963 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:00:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: audit 2023-11-17T05:59:59.367630+0000 mon.smithi006 (mon.0) 964 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:00.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: audit 2023-11-17T05:59:59.377267+0000 mon.smithi006 (mon.0) 965 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:00:00.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: audit 2023-11-17T05:59:59.389295+0000 mon.smithi006 (mon.0) 966 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:00.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:00 smithi156 bash[29655]: cluster 2023-11-17T06:00:00.000131+0000 mon.smithi006 (mon.0) 967 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T06:00:01.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:01 smithi006 bash[26022]: cluster 2023-11-17T05:59:59.368923+0000 mgr.smithi006.odexej (mgr.14176) 1224 : cluster [DBG] pgmap v843: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:01.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:01 smithi006 bash[26022]: cephadm 2023-11-17T05:59:59.393330+0000 mgr.smithi006.odexej (mgr.14176) 1225 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.gnrpci on smithi156 2023-11-17T06:00:01.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:01 smithi006 bash[26022]: cluster 2023-11-17T06:00:00.362924+0000 mon.smithi006 (mon.0) 968 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:00:01.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:01 smithi006 bash[26022]: cluster 2023-11-17T06:00:00.363010+0000 mon.smithi006 (mon.0) 969 : cluster [INF] Cluster is now healthy 2023-11-17T06:00:01.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:01 smithi156 bash[29655]: cluster 2023-11-17T05:59:59.368923+0000 mgr.smithi006.odexej (mgr.14176) 1224 : cluster [DBG] pgmap v843: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:01.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:01 smithi156 bash[29655]: cephadm 2023-11-17T05:59:59.393330+0000 mgr.smithi006.odexej (mgr.14176) 1225 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.gnrpci on smithi156 2023-11-17T06:00:01.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:01 smithi156 bash[29655]: cluster 2023-11-17T06:00:00.362924+0000 mon.smithi006 (mon.0) 968 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:00:01.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:01 smithi156 bash[29655]: cluster 2023-11-17T06:00:00.363010+0000 mon.smithi006 (mon.0) 969 : cluster [INF] Cluster is now healthy 2023-11-17T06:00:01.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:01.479 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:57:51.184874Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tozcch on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:02.065 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:03.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:03.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:03 smithi006 bash[26022]: cluster 2023-11-17T06:00:01.369704+0000 mgr.smithi006.odexej (mgr.14176) 1226 : cluster [DBG] pgmap v844: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:03.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:03 smithi006 bash[26022]: audit 2023-11-17T06:00:01.470393+0000 mgr.smithi006.odexej (mgr.14176) 1227 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:03.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:03 smithi156 bash[29655]: cluster 2023-11-17T06:00:01.369704+0000 mgr.smithi006.odexej (mgr.14176) 1226 : cluster [DBG] pgmap v844: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:03.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:03 smithi156 bash[29655]: audit 2023-11-17T06:00:01.470393+0000 mgr.smithi006.odexej (mgr.14176) 1227 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:05.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:05 smithi006 bash[26022]: cluster 2023-11-17T06:00:03.370593+0000 mgr.smithi006.odexej (mgr.14176) 1228 : cluster [DBG] pgmap v845: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:05.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:05 smithi006 bash[26022]: audit 2023-11-17T06:00:04.200859+0000 mon.smithi006 (mon.0) 970 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.gnrpci"}]: dispatch 2023-11-17T06:00:05.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:05 smithi006 bash[26022]: audit 2023-11-17T06:00:04.732633+0000 mon.smithi006 (mon.0) 971 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:05.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:05 smithi156 bash[29655]: cluster 2023-11-17T06:00:03.370593+0000 mgr.smithi006.odexej (mgr.14176) 1228 : cluster [DBG] pgmap v845: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:05.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:05 smithi156 bash[29655]: audit 2023-11-17T06:00:04.200859+0000 mon.smithi006 (mon.0) 970 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.gnrpci"}]: dispatch 2023-11-17T06:00:05.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:05 smithi156 bash[29655]: audit 2023-11-17T06:00:04.732633+0000 mon.smithi006 (mon.0) 971 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:06.392 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:06.392 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:55.295907Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.egadkh on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-egadkh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.egadkh\nDeploy daemon haproxy.nfs.foo.smithi156.egadkh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.028289Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xmrwce on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xmrwce\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xmrwce\nDeploy daemon haproxy.nfs.foo.smithi006.xmrwce ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T05:58:59.033037Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fqzkik on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:06.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: cephadm 2023-11-17T06:00:04.199403+0000 mgr.smithi006.odexej (mgr.14176) 1229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:00:06.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: cephadm 2023-11-17T06:00:04.200170+0000 mgr.smithi006.odexej (mgr.14176) 1230 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: cephadm 2023-11-17T06:00:04.202558+0000 mgr.smithi006.odexej (mgr.14176) 1231 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:06 smithi006 bash[26022]: cephadm 2023-11-17T06:00:04.207834+0000 mgr.smithi006.odexej (mgr.14176) 1232 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.dntkqu on smithi006 2023-11-17T06:00:06.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: cephadm 2023-11-17T06:00:04.199403+0000 mgr.smithi006.odexej (mgr.14176) 1229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: cephadm 2023-11-17T06:00:04.200170+0000 mgr.smithi006.odexej (mgr.14176) 1230 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: cephadm 2023-11-17T06:00:04.202558+0000 mgr.smithi006.odexej (mgr.14176) 1231 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:06.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci 2023-11-17T06:00:06.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.gnrpci ... 2023-11-17T06:00:06.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:06.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:06.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:06.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:06.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:06 smithi156 bash[29655]: cephadm 2023-11-17T06:00:04.207834+0000 mgr.smithi006.odexej (mgr.14176) 1232 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.dntkqu on smithi006 2023-11-17T06:00:07.024 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:07.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:07 smithi006 bash[26022]: cluster 2023-11-17T06:00:05.371728+0000 mgr.smithi006.odexej (mgr.14176) 1233 : cluster [DBG] pgmap v846: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:07.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:07 smithi156 bash[29655]: cluster 2023-11-17T06:00:05.371728+0000 mgr.smithi006.odexej (mgr.14176) 1233 : cluster [DBG] pgmap v846: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:08.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:08.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:08 smithi006 bash[26022]: audit 2023-11-17T06:00:06.376406+0000 mgr.smithi006.odexej (mgr.14176) 1234 : audit [DBG] from='client.15154 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:08.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:08 smithi006 bash[26022]: audit 2023-11-17T06:00:08.021544+0000 mon.smithi006 (mon.0) 972 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.dntkqu"}]: dispatch 2023-11-17T06:00:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:08 smithi156 bash[29655]: audit 2023-11-17T06:00:06.376406+0000 mgr.smithi006.odexej (mgr.14176) 1234 : audit [DBG] from='client.15154 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:08 smithi156 bash[29655]: audit 2023-11-17T06:00:08.021544+0000 mon.smithi006 (mon.0) 972 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.dntkqu"}]: dispatch 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cluster 2023-11-17T06:00:07.372622+0000 mgr.smithi006.odexej (mgr.14176) 1235 : cluster [DBG] pgmap v847: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cephadm 2023-11-17T06:00:08.019836+0000 mgr.smithi006.odexej (mgr.14176) 1236 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cephadm 2023-11-17T06:00:08.020370+0000 mgr.smithi006.odexej (mgr.14176) 1237 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cephadm 2023-11-17T06:00:08.022976+0000 mgr.smithi006.odexej (mgr.14176) 1238 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cephadm 2023-11-17T06:00:08.027825+0000 mgr.smithi006.odexej (mgr.14176) 1239 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cephadm 2023-11-17T06:00:08.031560+0000 mgr.smithi006.odexej (mgr.14176) 1240 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:00:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cluster 2023-11-17T06:00:08.032621+0000 mgr.smithi006.odexej (mgr.14176) 1241 : cluster [DBG] pgmap v848: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-11-17T06:00:09.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:09 smithi156 bash[29655]: cluster 2023-11-17T06:00:08.161584+0000 mon.smithi006 (mon.0) 973 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cluster 2023-11-17T06:00:07.372622+0000 mgr.smithi006.odexej (mgr.14176) 1235 : cluster [DBG] pgmap v847: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cephadm 2023-11-17T06:00:08.019836+0000 mgr.smithi006.odexej (mgr.14176) 1236 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.518 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.520 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cephadm 2023-11-17T06:00:08.020370+0000 mgr.smithi006.odexej (mgr.14176) 1237 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cephadm 2023-11-17T06:00:08.022976+0000 mgr.smithi006.odexej (mgr.14176) 1238 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.dntkqu ... 2023-11-17T06:00:09.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cephadm 2023-11-17T06:00:08.027825+0000 mgr.smithi006.odexej (mgr.14176) 1239 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cephadm 2023-11-17T06:00:08.031560+0000 mgr.smithi006.odexej (mgr.14176) 1240 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cluster 2023-11-17T06:00:08.032621+0000 mgr.smithi006.odexej (mgr.14176) 1241 : cluster [DBG] pgmap v848: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 114 B/s rd, 0 op/s 2023-11-17T06:00:09.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:09 smithi006 bash[26022]: cluster 2023-11-17T06:00:08.161584+0000 mon.smithi006 (mon.0) 973 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:00:11.268 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:11.268 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:11.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:11 smithi006 bash[26022]: cluster 2023-11-17T06:00:10.034018+0000 mgr.smithi006.odexej (mgr.14176) 1242 : cluster [DBG] pgmap v849: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:11.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:11 smithi006 bash[26022]: audit 2023-11-17T06:00:10.936357+0000 mon.smithi006 (mon.0) 974 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:11.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:11 smithi156 bash[29655]: cluster 2023-11-17T06:00:10.034018+0000 mgr.smithi006.odexej (mgr.14176) 1242 : cluster [DBG] pgmap v849: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:11.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:11 smithi156 bash[29655]: audit 2023-11-17T06:00:10.936357+0000 mon.smithi006 (mon.0) 974 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:11.863 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:12.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:13.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:12 smithi006 bash[26022]: audit 2023-11-17T06:00:11.259058+0000 mgr.smithi006.odexej (mgr.14176) 1243 : audit [DBG] from='client.15158 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:13.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:12 smithi006 bash[26022]: cluster 2023-11-17T06:00:12.035198+0000 mgr.smithi006.odexej (mgr.14176) 1244 : cluster [DBG] pgmap v850: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:12 smithi156 bash[29655]: audit 2023-11-17T06:00:11.259058+0000 mgr.smithi006.odexej (mgr.14176) 1243 : audit [DBG] from='client.15158 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:12 smithi156 bash[29655]: cluster 2023-11-17T06:00:12.035198+0000 mgr.smithi006.odexej (mgr.14176) 1244 : cluster [DBG] pgmap v850: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:15.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:15 smithi156 bash[29655]: cluster 2023-11-17T06:00:14.036452+0000 mgr.smithi006.odexej (mgr.14176) 1245 : cluster [DBG] pgmap v851: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:15.442 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:15 smithi006 bash[26022]: cluster 2023-11-17T06:00:14.036452+0000 mgr.smithi006.odexej (mgr.14176) 1245 : cluster [DBG] pgmap v851: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:16.333 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:16.333 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:16.917 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:17.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:17 smithi006 bash[26022]: cluster 2023-11-17T06:00:16.037183+0000 mgr.smithi006.odexej (mgr.14176) 1246 : cluster [DBG] pgmap v852: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:17.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:17 smithi156 bash[29655]: cluster 2023-11-17T06:00:16.037183+0000 mgr.smithi006.odexej (mgr.14176) 1246 : cluster [DBG] pgmap v852: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:17.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:20.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:18 smithi156 bash[29655]: audit 2023-11-17T06:00:16.318850+0000 mgr.smithi006.odexej (mgr.14176) 1247 : audit [DBG] from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:20.242 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:19 smithi156 bash[29655]: cluster 2023-11-17T06:00:18.038155+0000 mgr.smithi006.odexej (mgr.14176) 1248 : cluster [DBG] pgmap v853: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:20.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:18 smithi006 bash[26022]: audit 2023-11-17T06:00:16.318850+0000 mgr.smithi006.odexej (mgr.14176) 1247 : audit [DBG] from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:20.242 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:19 smithi006 bash[26022]: cluster 2023-11-17T06:00:18.038155+0000 mgr.smithi006.odexej (mgr.14176) 1248 : cluster [DBG] pgmap v853: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:00:21.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:20 smithi006 bash[26022]: audit 2023-11-17T06:00:19.733494+0000 mon.smithi006 (mon.0) 975 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:21.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:20 smithi006 bash[26022]: cluster 2023-11-17T06:00:20.039492+0000 mgr.smithi006.odexej (mgr.14176) 1249 : cluster [DBG] pgmap v854: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:21.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:20 smithi156 bash[29655]: audit 2023-11-17T06:00:19.733494+0000 mon.smithi006 (mon.0) 975 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:00:21.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:20 smithi156 bash[29655]: cluster 2023-11-17T06:00:20.039492+0000 mgr.smithi006.odexej (mgr.14176) 1249 : cluster [DBG] pgmap v854: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:22.953 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:22 smithi006 bash[26022]: cluster 2023-11-17T06:00:22.040531+0000 mgr.smithi006.odexej (mgr.14176) 1250 : cluster [DBG] pgmap v855: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:23.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:22 smithi156 bash[29655]: cluster 2023-11-17T06:00:22.040531+0000 mgr.smithi006.odexej (mgr.14176) 1250 : cluster [DBG] pgmap v855: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:23.405 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:23.406 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:24.058 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:25.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:25 smithi006 bash[26022]: audit 2023-11-17T06:00:23.388236+0000 mgr.smithi006.odexej (mgr.14176) 1251 : audit [DBG] from='client.15166 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:25 smithi006 bash[26022]: cluster 2023-11-17T06:00:24.041452+0000 mgr.smithi006.odexej (mgr.14176) 1252 : cluster [DBG] pgmap v856: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:25 smithi156 bash[29655]: audit 2023-11-17T06:00:23.388236+0000 mgr.smithi006.odexej (mgr.14176) 1251 : audit [DBG] from='client.15166 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:25 smithi156 bash[29655]: cluster 2023-11-17T06:00:24.041452+0000 mgr.smithi006.odexej (mgr.14176) 1252 : cluster [DBG] pgmap v856: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:27.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:27 smithi006 bash[26022]: cluster 2023-11-17T06:00:26.042317+0000 mgr.smithi006.odexej (mgr.14176) 1253 : cluster [DBG] pgmap v857: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:44.940 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:44.941 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:44.942 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:45.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:27 smithi156 bash[29655]: cluster 2023-11-17T06:00:26.042317+0000 mgr.smithi006.odexej (mgr.14176) 1253 : cluster [DBG] pgmap v857: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.062 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:29 smithi156 bash[29655]: cluster 2023-11-17T06:00:28.043426+0000 mgr.smithi006.odexej (mgr.14176) 1254 : cluster [DBG] pgmap v858: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:29 smithi156 bash[29655]: audit 2023-11-17T06:00:28.115398+0000 mgr.smithi006.odexej (mgr.14176) 1255 : audit [DBG] from='client.15170 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:31 smithi156 bash[29655]: cluster 2023-11-17T06:00:30.044851+0000 mgr.smithi006.odexej (mgr.14176) 1256 : cluster [DBG] pgmap v859: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:32 smithi156 bash[29655]: cluster 2023-11-17T06:00:32.045817+0000 mgr.smithi006.odexej (mgr.14176) 1257 : cluster [DBG] pgmap v860: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:35 smithi156 bash[29655]: cluster 2023-11-17T06:00:34.046949+0000 mgr.smithi006.odexej (mgr.14176) 1258 : cluster [DBG] pgmap v861: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:37 smithi156 bash[29655]: cluster 2023-11-17T06:00:36.048300+0000 mgr.smithi006.odexej (mgr.14176) 1259 : cluster [DBG] pgmap v862: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:39 smithi156 bash[29655]: cluster 2023-11-17T06:00:38.049342+0000 mgr.smithi006.odexej (mgr.14176) 1260 : cluster [DBG] pgmap v863: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:41 smithi156 bash[29655]: cluster 2023-11-17T06:00:40.050792+0000 mgr.smithi006.odexej (mgr.14176) 1261 : cluster [DBG] pgmap v864: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.063 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:42 smithi156 bash[29655]: cluster 2023-11-17T06:00:42.051975+0000 mgr.smithi006.odexej (mgr.14176) 1262 : cluster [DBG] pgmap v865: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:29 smithi006 bash[26022]: cluster 2023-11-17T06:00:28.043426+0000 mgr.smithi006.odexej (mgr.14176) 1254 : cluster [DBG] pgmap v858: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:29 smithi006 bash[26022]: audit 2023-11-17T06:00:28.115398+0000 mgr.smithi006.odexej (mgr.14176) 1255 : audit [DBG] from='client.15170 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:31 smithi006 bash[26022]: cluster 2023-11-17T06:00:30.044851+0000 mgr.smithi006.odexej (mgr.14176) 1256 : cluster [DBG] pgmap v859: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:32 smithi006 bash[26022]: cluster 2023-11-17T06:00:32.045817+0000 mgr.smithi006.odexej (mgr.14176) 1257 : cluster [DBG] pgmap v860: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:35 smithi006 bash[26022]: cluster 2023-11-17T06:00:34.046949+0000 mgr.smithi006.odexej (mgr.14176) 1258 : cluster [DBG] pgmap v861: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:37 smithi006 bash[26022]: cluster 2023-11-17T06:00:36.048300+0000 mgr.smithi006.odexej (mgr.14176) 1259 : cluster [DBG] pgmap v862: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:39 smithi006 bash[26022]: cluster 2023-11-17T06:00:38.049342+0000 mgr.smithi006.odexej (mgr.14176) 1260 : cluster [DBG] pgmap v863: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:41 smithi006 bash[26022]: cluster 2023-11-17T06:00:40.050792+0000 mgr.smithi006.odexej (mgr.14176) 1261 : cluster [DBG] pgmap v864: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:42 smithi006 bash[26022]: cluster 2023-11-17T06:00:42.051975+0000 mgr.smithi006.odexej (mgr.14176) 1262 : cluster [DBG] pgmap v865: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:45.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:45 smithi006 bash[26022]: cluster 2023-11-17T06:00:44.053323+0000 mgr.smithi006.odexej (mgr.14176) 1263 : cluster [DBG] pgmap v866: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:45.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:45 smithi156 bash[29655]: cluster 2023-11-17T06:00:44.053323+0000 mgr.smithi006.odexej (mgr.14176) 1263 : cluster [DBG] pgmap v866: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:46.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:47.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:47 smithi006 bash[26022]: cluster 2023-11-17T06:00:46.054730+0000 mgr.smithi006.odexej (mgr.14176) 1264 : cluster [DBG] pgmap v867: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:47.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:47 smithi156 bash[29655]: cluster 2023-11-17T06:00:46.054730+0000 mgr.smithi006.odexej (mgr.14176) 1264 : cluster [DBG] pgmap v867: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:49.118 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:49.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:49 smithi006 bash[26022]: cluster 2023-11-17T06:00:48.055820+0000 mgr.smithi006.odexej (mgr.14176) 1265 : cluster [DBG] pgmap v868: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:49.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:49 smithi156 bash[29655]: cluster 2023-11-17T06:00:48.055820+0000 mgr.smithi006.odexej (mgr.14176) 1265 : cluster [DBG] pgmap v868: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:49.719 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:50.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:50 smithi006 bash[26022]: audit 2023-11-17T06:00:49.100807+0000 mgr.smithi006.odexej (mgr.14176) 1266 : audit [DBG] from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:50.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:50 smithi156 bash[29655]: audit 2023-11-17T06:00:49.100807+0000 mgr.smithi006.odexej (mgr.14176) 1266 : audit [DBG] from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:50.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:51.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:51 smithi006 bash[26022]: cluster 2023-11-17T06:00:50.056751+0000 mgr.smithi006.odexej (mgr.14176) 1267 : cluster [DBG] pgmap v869: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:51.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:51 smithi156 bash[29655]: cluster 2023-11-17T06:00:50.056751+0000 mgr.smithi006.odexej (mgr.14176) 1267 : cluster [DBG] pgmap v869: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:53.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:52 smithi006 bash[26022]: cluster 2023-11-17T06:00:52.057743+0000 mgr.smithi006.odexej (mgr.14176) 1268 : cluster [DBG] pgmap v870: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:53.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:52 smithi156 bash[29655]: cluster 2023-11-17T06:00:52.057743+0000 mgr.smithi006.odexej (mgr.14176) 1268 : cluster [DBG] pgmap v870: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:53.968 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:53.969 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:54.600 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:55.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:55 smithi006 bash[26022]: audit 2023-11-17T06:00:53.957821+0000 mgr.smithi006.odexej (mgr.14176) 1269 : audit [DBG] from='client.15178 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:55.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:55 smithi006 bash[26022]: cluster 2023-11-17T06:00:54.058524+0000 mgr.smithi006.odexej (mgr.14176) 1270 : cluster [DBG] pgmap v871: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:55.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:55 smithi156 bash[29655]: audit 2023-11-17T06:00:53.957821+0000 mgr.smithi006.odexej (mgr.14176) 1269 : audit [DBG] from='client.15178 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:00:55.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:55 smithi156 bash[29655]: cluster 2023-11-17T06:00:54.058524+0000 mgr.smithi006.odexej (mgr.14176) 1270 : cluster [DBG] pgmap v871: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:00:55.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:00:57.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:57 smithi006 bash[26022]: cluster 2023-11-17T06:00:56.059860+0000 mgr.smithi006.odexej (mgr.14176) 1271 : cluster [DBG] pgmap v872: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:57.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:57 smithi156 bash[29655]: cluster 2023-11-17T06:00:56.059860+0000 mgr.smithi006.odexej (mgr.14176) 1271 : cluster [DBG] pgmap v872: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:58.739 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:00:58.739 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:00:59.353 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:00:59.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:00:59 smithi006 bash[26022]: cluster 2023-11-17T06:00:58.060518+0000 mgr.smithi006.odexej (mgr.14176) 1272 : cluster [DBG] pgmap v873: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:00:59.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:00:59 smithi156 bash[29655]: cluster 2023-11-17T06:00:58.060518+0000 mgr.smithi006.odexej (mgr.14176) 1272 : cluster [DBG] pgmap v873: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:00.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:00.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:00 smithi006 bash[26022]: audit 2023-11-17T06:00:58.724598+0000 mgr.smithi006.odexej (mgr.14176) 1273 : audit [DBG] from='client.15182 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:00.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:00 smithi156 bash[29655]: audit 2023-11-17T06:00:58.724598+0000 mgr.smithi006.odexej (mgr.14176) 1273 : audit [DBG] from='client.15182 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:01.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:01 smithi006 bash[26022]: cluster 2023-11-17T06:01:00.061916+0000 mgr.smithi006.odexej (mgr.14176) 1274 : cluster [DBG] pgmap v874: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:01.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:01 smithi156 bash[29655]: cluster 2023-11-17T06:01:00.061916+0000 mgr.smithi006.odexej (mgr.14176) 1274 : cluster [DBG] pgmap v874: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:03.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:02 smithi006 bash[26022]: cluster 2023-11-17T06:01:02.062824+0000 mgr.smithi006.odexej (mgr.14176) 1275 : cluster [DBG] pgmap v875: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:03.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:02 smithi156 bash[29655]: cluster 2023-11-17T06:01:02.062824+0000 mgr.smithi006.odexej (mgr.14176) 1275 : cluster [DBG] pgmap v875: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:03.235 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:03.235 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:03.863 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:04.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:05.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:05 smithi006 bash[26022]: audit 2023-11-17T06:01:03.218918+0000 mgr.smithi006.odexej (mgr.14176) 1276 : audit [DBG] from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:05.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:05 smithi006 bash[26022]: cluster 2023-11-17T06:01:04.064019+0000 mgr.smithi006.odexej (mgr.14176) 1277 : cluster [DBG] pgmap v876: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:05.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:05 smithi156 bash[29655]: audit 2023-11-17T06:01:03.218918+0000 mgr.smithi006.odexej (mgr.14176) 1276 : audit [DBG] from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:05.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:05 smithi156 bash[29655]: cluster 2023-11-17T06:01:04.064019+0000 mgr.smithi006.odexej (mgr.14176) 1277 : cluster [DBG] pgmap v876: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:07.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:07 smithi006 bash[26022]: cluster 2023-11-17T06:01:06.065431+0000 mgr.smithi006.odexej (mgr.14176) 1278 : cluster [DBG] pgmap v877: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:07.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:07 smithi156 bash[29655]: cluster 2023-11-17T06:01:06.065431+0000 mgr.smithi006.odexej (mgr.14176) 1278 : cluster [DBG] pgmap v877: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:08.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:08.187 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T05:58:59.036418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.vxfhjr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:08.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:08 smithi006 bash[26022]: audit 2023-11-17T06:01:08.035852+0000 mon.smithi006 (mon.0) 976 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:01:08.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:08 smithi156 bash[29655]: audit 2023-11-17T06:01:08.035852+0000 mon.smithi006 (mon.0) 976 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:01:08.814 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: cluster 2023-11-17T06:01:08.066394+0000 mgr.smithi006.odexej (mgr.14176) 1279 : cluster [DBG] pgmap v878: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.174448+0000 mgr.smithi006.odexej (mgr.14176) 1280 : audit [DBG] from='client.15190 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.341539+0000 mon.smithi006 (mon.0) 977 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.342445+0000 mon.smithi006 (mon.0) 978 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.347656+0000 mon.smithi006 (mon.0) 979 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.354095+0000 mon.smithi006 (mon.0) 980 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:01:09.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:09 smithi006 bash[26022]: audit 2023-11-17T06:01:08.366259+0000 mon.smithi006 (mon.0) 981 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:09.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: cluster 2023-11-17T06:01:08.066394+0000 mgr.smithi006.odexej (mgr.14176) 1279 : cluster [DBG] pgmap v878: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.174448+0000 mgr.smithi006.odexej (mgr.14176) 1280 : audit [DBG] from='client.15190 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.341539+0000 mon.smithi006 (mon.0) 977 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.342445+0000 mon.smithi006 (mon.0) 978 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.347656+0000 mon.smithi006 (mon.0) 979 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.354095+0000 mon.smithi006 (mon.0) 980 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:01:09.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:09 smithi156 bash[29655]: audit 2023-11-17T06:01:08.366259+0000 mon.smithi006 (mon.0) 981 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:09.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:10.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:10 smithi156 bash[29655]: cluster 2023-11-17T06:01:08.348355+0000 mgr.smithi006.odexej (mgr.14176) 1281 : cluster [DBG] pgmap v879: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:26.334 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:10 smithi156 bash[29655]: cephadm 2023-11-17T06:01:08.370044+0000 mgr.smithi006.odexej (mgr.14176) 1282 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ykjwzu on smithi156 2023-11-17T06:01:26.337 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:10 smithi156 bash[29655]: cluster 2023-11-17T06:01:09.345825+0000 mon.smithi006 (mon.0) 982 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:01:26.342 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:10 smithi156 bash[29655]: cluster 2023-11-17T06:01:09.345881+0000 mon.smithi006 (mon.0) 983 : cluster [INF] Cluster is now healthy 2023-11-17T06:01:26.364 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:10 smithi006 bash[26022]: cluster 2023-11-17T06:01:08.348355+0000 mgr.smithi006.odexej (mgr.14176) 1281 : cluster [DBG] pgmap v879: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:26.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:10 smithi006 bash[26022]: cephadm 2023-11-17T06:01:08.370044+0000 mgr.smithi006.odexej (mgr.14176) 1282 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ykjwzu on smithi156 2023-11-17T06:01:26.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:10 smithi006 bash[26022]: cluster 2023-11-17T06:01:09.345825+0000 mon.smithi006 (mon.0) 982 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:01:26.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:10 smithi006 bash[26022]: cluster 2023-11-17T06:01:09.345881+0000 mon.smithi006 (mon.0) 983 : cluster [INF] Cluster is now healthy 2023-11-17T06:01:26.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:12 smithi006 bash[26022]: cluster 2023-11-17T06:01:10.349395+0000 mgr.smithi006.odexej (mgr.14176) 1283 : cluster [DBG] pgmap v880: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:26.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:12 smithi006 bash[26022]: audit 2023-11-17T06:01:12.137142+0000 mon.smithi006 (mon.0) 984 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ykjwzu"}]: dispatch 2023-11-17T06:01:26.523 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:26.523 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:04.202294Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.gnrpci on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-gnrpci\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.gnrpci\nDeploy daemon haproxy.nfs.foo.smithi156.gnrpci ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.022824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.dntkqu on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-dntkqu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.dntkqu\nDeploy daemon haproxy.nfs.foo.smithi006.dntkqu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:00:08.027600Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.irtttg on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:26.524 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: cephadm 2023-11-17T06:01:12.135968+0000 mgr.smithi006.odexej (mgr.14176) 1284 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.524 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.524 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.524 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:01:26.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:26.527 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:26.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:26.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: cephadm 2023-11-17T06:01:12.136462+0000 mgr.smithi006.odexej (mgr.14176) 1285 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: cephadm 2023-11-17T06:01:12.138790+0000 mgr.smithi006.odexej (mgr.14176) 1286 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:26.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:26.573 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.608 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:13 smithi006 bash[26022]: cephadm 2023-11-17T06:01:12.143937+0000 mgr.smithi006.odexej (mgr.14176) 1287 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.lbcerv on smithi006 2023-11-17T06:01:26.610 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:12 smithi156 bash[29655]: cluster 2023-11-17T06:01:10.349395+0000 mgr.smithi006.odexej (mgr.14176) 1283 : cluster [DBG] pgmap v880: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:26.610 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:12 smithi156 bash[29655]: audit 2023-11-17T06:01:12.137142+0000 mon.smithi006 (mon.0) 984 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ykjwzu"}]: dispatch 2023-11-17T06:01:26.610 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: cephadm 2023-11-17T06:01:12.135968+0000 mgr.smithi006.odexej (mgr.14176) 1284 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:26.611 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:26.612 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:01:26.940 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:01:26.940 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:01:26.940 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:01:26.940 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:01:26.941 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:26.941 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:26.941 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:27.005 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: cephadm 2023-11-17T06:01:12.136462+0000 mgr.smithi006.odexej (mgr.14176) 1285 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: cephadm 2023-11-17T06:01:12.138790+0000 mgr.smithi006.odexej (mgr.14176) 1286 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu 2023-11-17T06:01:27.043 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ykjwzu ... 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:13 smithi156 bash[29655]: cephadm 2023-11-17T06:01:12.143937+0000 mgr.smithi006.odexej (mgr.14176) 1287 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.lbcerv on smithi006 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:14 smithi156 bash[29655]: cluster 2023-11-17T06:01:12.349922+0000 mgr.smithi006.odexej (mgr.14176) 1288 : cluster [DBG] pgmap v881: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:14 smithi156 bash[29655]: audit 2023-11-17T06:01:12.780972+0000 mgr.smithi006.odexej (mgr.14176) 1289 : audit [DBG] from='client.15194 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:16 smithi156 bash[29655]: cluster 2023-11-17T06:01:14.350929+0000 mgr.smithi006.odexej (mgr.14176) 1290 : cluster [DBG] pgmap v882: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:27.044 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:16 smithi156 bash[29655]: audit 2023-11-17T06:01:15.905033+0000 mon.smithi006 (mon.0) 985 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.lbcerv"}]: dispatch 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:16 smithi156 bash[29655]: audit 2023-11-17T06:01:15.982886+0000 mon.smithi006 (mon.0) 986 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cephadm 2023-11-17T06:01:15.903912+0000 mgr.smithi006.odexej (mgr.14176) 1291 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.045 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cephadm 2023-11-17T06:01:15.904360+0000 mgr.smithi006.odexej (mgr.14176) 1292 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cephadm 2023-11-17T06:01:15.906584+0000 mgr.smithi006.odexej (mgr.14176) 1293 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cephadm 2023-11-17T06:01:15.910616+0000 mgr.smithi006.odexej (mgr.14176) 1294 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cephadm 2023-11-17T06:01:15.914254+0000 mgr.smithi006.odexej (mgr.14176) 1295 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:01:27.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cluster 2023-11-17T06:01:15.915980+0000 mgr.smithi006.odexej (mgr.14176) 1296 : cluster [DBG] pgmap v883: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T06:01:27.092 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:17 smithi156 bash[29655]: cluster 2023-11-17T06:01:16.165694+0000 mon.smithi006 (mon.0) 987 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:01:27.092 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:19 smithi156 bash[29655]: cluster 2023-11-17T06:01:17.917120+0000 mgr.smithi006.odexej (mgr.14176) 1297 : cluster [DBG] pgmap v884: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T06:01:27.092 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:21 smithi156 bash[29655]: cluster 2023-11-17T06:01:19.918571+0000 mgr.smithi006.odexej (mgr.14176) 1298 : cluster [DBG] pgmap v885: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.092 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:22 smithi156 bash[29655]: cluster 2023-11-17T06:01:21.919618+0000 mgr.smithi006.odexej (mgr.14176) 1299 : cluster [DBG] pgmap v886: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.092 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:24 smithi156 bash[29655]: cluster 2023-11-17T06:01:23.920795+0000 mgr.smithi006.odexej (mgr.14176) 1300 : cluster [DBG] pgmap v887: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.093 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:14 smithi006 bash[26022]: cluster 2023-11-17T06:01:12.349922+0000 mgr.smithi006.odexej (mgr.14176) 1288 : cluster [DBG] pgmap v881: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:14 smithi006 bash[26022]: audit 2023-11-17T06:01:12.780972+0000 mgr.smithi006.odexej (mgr.14176) 1289 : audit [DBG] from='client.15194 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:16 smithi006 bash[26022]: cluster 2023-11-17T06:01:14.350929+0000 mgr.smithi006.odexej (mgr.14176) 1290 : cluster [DBG] pgmap v882: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:16 smithi006 bash[26022]: audit 2023-11-17T06:01:15.905033+0000 mon.smithi006 (mon.0) 985 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.lbcerv"}]: dispatch 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:16 smithi006 bash[26022]: audit 2023-11-17T06:01:15.982886+0000 mon.smithi006 (mon.0) 986 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cephadm 2023-11-17T06:01:15.903912+0000 mgr.smithi006.odexej (mgr.14176) 1291 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cephadm 2023-11-17T06:01:15.904360+0000 mgr.smithi006.odexej (mgr.14176) 1292 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cephadm 2023-11-17T06:01:15.906584+0000 mgr.smithi006.odexej (mgr.14176) 1293 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.lbcerv ... 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:01:27.097 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:01:27.334 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:01:27.334 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cephadm 2023-11-17T06:01:15.910616+0000 mgr.smithi006.odexej (mgr.14176) 1294 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:01:27.451 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cephadm 2023-11-17T06:01:15.914254+0000 mgr.smithi006.odexej (mgr.14176) 1295 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:01:27.451 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cluster 2023-11-17T06:01:15.915980+0000 mgr.smithi006.odexej (mgr.14176) 1296 : cluster [DBG] pgmap v883: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:17 smithi006 bash[26022]: cluster 2023-11-17T06:01:16.165694+0000 mon.smithi006 (mon.0) 987 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:19 smithi006 bash[26022]: cluster 2023-11-17T06:01:17.917120+0000 mgr.smithi006.odexej (mgr.14176) 1297 : cluster [DBG] pgmap v884: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 103 B/s rd, 0 op/s 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:21 smithi006 bash[26022]: cluster 2023-11-17T06:01:19.918571+0000 mgr.smithi006.odexej (mgr.14176) 1298 : cluster [DBG] pgmap v885: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:22 smithi006 bash[26022]: cluster 2023-11-17T06:01:21.919618+0000 mgr.smithi006.odexej (mgr.14176) 1299 : cluster [DBG] pgmap v886: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:24 smithi006 bash[26022]: cluster 2023-11-17T06:01:23.920795+0000 mgr.smithi006.odexej (mgr.14176) 1300 : cluster [DBG] pgmap v887: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:26 smithi006 bash[26022]: cluster 2023-11-17T06:01:25.922130+0000 mgr.smithi006.odexej (mgr.14176) 1301 : cluster [DBG] pgmap v888: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:27.453 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:26 smithi156 bash[29655]: cluster 2023-11-17T06:01:25.922130+0000 mgr.smithi006.odexej (mgr.14176) 1301 : cluster [DBG] pgmap v888: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:01:28.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:29.283 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:28 smithi006 bash[26022]: cluster 2023-11-17T06:01:27.923154+0000 mgr.smithi006.odexej (mgr.14176) 1302 : cluster [DBG] pgmap v889: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:29.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:28 smithi156 bash[29655]: cluster 2023-11-17T06:01:27.923154+0000 mgr.smithi006.odexej (mgr.14176) 1302 : cluster [DBG] pgmap v889: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:31.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:31.150 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:31.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:30 smithi006 bash[26022]: cluster 2023-11-17T06:01:29.924675+0000 mgr.smithi006.odexej (mgr.14176) 1303 : cluster [DBG] pgmap v890: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:31.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:30 smithi156 bash[29655]: cluster 2023-11-17T06:01:29.924675+0000 mgr.smithi006.odexej (mgr.14176) 1303 : cluster [DBG] pgmap v890: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:31.753 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:32.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:32 smithi006 bash[26022]: audit 2023-11-17T06:01:31.139187+0000 mgr.smithi006.odexej (mgr.14176) 1304 : audit [DBG] from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:32.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:32 smithi156 bash[29655]: audit 2023-11-17T06:01:31.139187+0000 mgr.smithi006.odexej (mgr.14176) 1304 : audit [DBG] from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:32.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:33.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:33 smithi006 bash[26022]: cluster 2023-11-17T06:01:31.925860+0000 mgr.smithi006.odexej (mgr.14176) 1305 : cluster [DBG] pgmap v891: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:33.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:33 smithi156 bash[29655]: cluster 2023-11-17T06:01:31.925860+0000 mgr.smithi006.odexej (mgr.14176) 1305 : cluster [DBG] pgmap v891: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:35.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:35 smithi006 bash[26022]: cluster 2023-11-17T06:01:33.926690+0000 mgr.smithi006.odexej (mgr.14176) 1306 : cluster [DBG] pgmap v892: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:35.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:35 smithi156 bash[29655]: cluster 2023-11-17T06:01:33.926690+0000 mgr.smithi006.odexej (mgr.14176) 1306 : cluster [DBG] pgmap v892: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:36.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:36.072 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:36.679 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:37.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:37 smithi006 bash[26022]: cluster 2023-11-17T06:01:35.928064+0000 mgr.smithi006.odexej (mgr.14176) 1307 : cluster [DBG] pgmap v893: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:37.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:37 smithi006 bash[26022]: audit 2023-11-17T06:01:36.056925+0000 mgr.smithi006.odexej (mgr.14176) 1308 : audit [DBG] from='client.15202 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:37.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:37 smithi156 bash[29655]: cluster 2023-11-17T06:01:35.928064+0000 mgr.smithi006.odexej (mgr.14176) 1307 : cluster [DBG] pgmap v893: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:37.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:37 smithi156 bash[29655]: audit 2023-11-17T06:01:36.056925+0000 mgr.smithi006.odexej (mgr.14176) 1308 : audit [DBG] from='client.15202 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:37.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:39.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:39 smithi006 bash[26022]: cluster 2023-11-17T06:01:37.929069+0000 mgr.smithi006.odexej (mgr.14176) 1309 : cluster [DBG] pgmap v894: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:39.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:39 smithi156 bash[29655]: cluster 2023-11-17T06:01:37.929069+0000 mgr.smithi006.odexej (mgr.14176) 1309 : cluster [DBG] pgmap v894: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:40.887 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:40.887 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:41.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:41 smithi006 bash[26022]: cluster 2023-11-17T06:01:39.930505+0000 mgr.smithi006.odexej (mgr.14176) 1310 : cluster [DBG] pgmap v895: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:41.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:41 smithi156 bash[29655]: cluster 2023-11-17T06:01:39.930505+0000 mgr.smithi006.odexej (mgr.14176) 1310 : cluster [DBG] pgmap v895: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:41.523 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:42.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:42 smithi006 bash[26022]: audit 2023-11-17T06:01:40.873109+0000 mgr.smithi006.odexej (mgr.14176) 1311 : audit [DBG] from='client.15206 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:42.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:42 smithi156 bash[29655]: audit 2023-11-17T06:01:40.873109+0000 mgr.smithi006.odexej (mgr.14176) 1311 : audit [DBG] from='client.15206 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:42.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:43.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:43 smithi006 bash[26022]: cluster 2023-11-17T06:01:41.931561+0000 mgr.smithi006.odexej (mgr.14176) 1312 : cluster [DBG] pgmap v896: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:43.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:43 smithi156 bash[29655]: cluster 2023-11-17T06:01:41.931561+0000 mgr.smithi006.odexej (mgr.14176) 1312 : cluster [DBG] pgmap v896: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:45.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:45 smithi006 bash[26022]: cluster 2023-11-17T06:01:43.932868+0000 mgr.smithi006.odexej (mgr.14176) 1313 : cluster [DBG] pgmap v897: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:45.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:45 smithi156 bash[29655]: cluster 2023-11-17T06:01:43.932868+0000 mgr.smithi006.odexej (mgr.14176) 1313 : cluster [DBG] pgmap v897: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:45.710 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:45.710 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:46.354 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:47.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:47.366 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:47 smithi006 bash[26022]: audit 2023-11-17T06:01:45.701416+0000 mgr.smithi006.odexej (mgr.14176) 1314 : audit [DBG] from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:47.367 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:47 smithi006 bash[26022]: cluster 2023-11-17T06:01:45.934337+0000 mgr.smithi006.odexej (mgr.14176) 1315 : cluster [DBG] pgmap v898: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:47.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:47 smithi156 bash[29655]: audit 2023-11-17T06:01:45.701416+0000 mgr.smithi006.odexej (mgr.14176) 1314 : audit [DBG] from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:47.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:47 smithi156 bash[29655]: cluster 2023-11-17T06:01:45.934337+0000 mgr.smithi006.odexej (mgr.14176) 1315 : cluster [DBG] pgmap v898: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:49.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:49 smithi006 bash[26022]: cluster 2023-11-17T06:01:47.935419+0000 mgr.smithi006.odexej (mgr.14176) 1316 : cluster [DBG] pgmap v899: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:49.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:49 smithi156 bash[29655]: cluster 2023-11-17T06:01:47.935419+0000 mgr.smithi006.odexej (mgr.14176) 1316 : cluster [DBG] pgmap v899: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:50.671 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:50.671 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:51.262 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:51.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:51 smithi006 bash[26022]: cluster 2023-11-17T06:01:49.936828+0000 mgr.smithi006.odexej (mgr.14176) 1317 : cluster [DBG] pgmap v900: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:51.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:51 smithi156 bash[29655]: cluster 2023-11-17T06:01:49.936828+0000 mgr.smithi006.odexej (mgr.14176) 1317 : cluster [DBG] pgmap v900: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:52.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:01:52.360 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:52 smithi006 bash[26022]: audit 2023-11-17T06:01:50.654282+0000 mgr.smithi006.odexej (mgr.14176) 1318 : audit [DBG] from='client.15214 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:52.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:52 smithi156 bash[29655]: audit 2023-11-17T06:01:50.654282+0000 mgr.smithi006.odexej (mgr.14176) 1318 : audit [DBG] from='client.15214 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:01:53.396 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:53 smithi006 bash[26022]: cluster 2023-11-17T06:01:51.937870+0000 mgr.smithi006.odexej (mgr.14176) 1319 : cluster [DBG] pgmap v901: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:53.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:53 smithi156 bash[29655]: cluster 2023-11-17T06:01:51.937870+0000 mgr.smithi006.odexej (mgr.14176) 1319 : cluster [DBG] pgmap v901: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:01:55.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:55 smithi006 bash[26022]: cluster 2023-11-17T06:01:53.939099+0000 mgr.smithi006.odexej (mgr.14176) 1320 : cluster [DBG] pgmap v902: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:01:55.346 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T05:52:02.416254Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T05:52:01.670232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T05:52:02.416592Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:00:08.031312Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.dscusm on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:12.138515Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ykjwzu on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ykjwzu\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ykjwzu\nDeploy daemon haproxy.nfs.foo.smithi156.ykjwzu ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.906347Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.lbcerv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-lbcerv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.lbcerv\nDeploy daemon haproxy.nfs.foo.smithi006.lbcerv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:01:15.910390Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ajbgho on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T05:52:01.671852Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T05:52:01.670518Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T05:52:01.670716Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T05:52:02.417928Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T05:52:01.670902Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T05:52:01.671085Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T05:52:02.416927Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:01:55.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:55 smithi156 bash[29655]: cluster 2023-11-17T06:01:53.939099+0000 mgr.smithi006.odexej (mgr.14176) 1320 : cluster [DBG] pgmap v902: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:01:55.986 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:01:56.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:07.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:57 smithi156 bash[29655]: audit 2023-11-17T06:01:55.329019+0000 mgr.smithi006.odexej (mgr.14176) 1321 : audit [DBG] from='client.15218 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:07.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:57 smithi156 bash[29655]: cluster 2023-11-17T06:01:55.940504+0000 mgr.smithi006.odexej (mgr.14176) 1322 : cluster [DBG] pgmap v903: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:01:59 smithi156 bash[29655]: cluster 2023-11-17T06:01:57.941494+0000 mgr.smithi006.odexej (mgr.14176) 1323 : cluster [DBG] pgmap v904: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:01 smithi156 bash[29655]: cluster 2023-11-17T06:01:59.942850+0000 mgr.smithi006.odexej (mgr.14176) 1324 : cluster [DBG] pgmap v905: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:02 smithi156 bash[29655]: cluster 2023-11-17T06:02:01.943887+0000 mgr.smithi006.odexej (mgr.14176) 1325 : cluster [DBG] pgmap v906: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:05 smithi156 bash[29655]: cluster 2023-11-17T06:02:03.945053+0000 mgr.smithi006.odexej (mgr.14176) 1326 : cluster [DBG] pgmap v907: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:07 smithi156 bash[29655]: cluster 2023-11-17T06:02:05.946385+0000 mgr.smithi006.odexej (mgr.14176) 1327 : cluster [DBG] pgmap v908: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:09 smithi156 bash[29655]: cluster 2023-11-17T06:02:07.947448+0000 mgr.smithi006.odexej (mgr.14176) 1328 : cluster [DBG] pgmap v909: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:11 smithi156 bash[29655]: cluster 2023-11-17T06:02:09.948899+0000 mgr.smithi006.odexej (mgr.14176) 1329 : cluster [DBG] pgmap v910: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:12 smithi156 bash[29655]: cluster 2023-11-17T06:02:11.950111+0000 mgr.smithi006.odexej (mgr.14176) 1330 : cluster [DBG] pgmap v911: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:15 smithi156 bash[29655]: cluster 2023-11-17T06:02:13.951271+0000 mgr.smithi006.odexej (mgr.14176) 1331 : cluster [DBG] pgmap v912: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:16 smithi156 bash[29655]: audit 2023-11-17T06:02:15.919172+0000 mon.smithi006 (mon.0) 988 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:17 smithi156 bash[29655]: cluster 2023-11-17T06:02:15.952431+0000 mgr.smithi006.odexej (mgr.14176) 1332 : cluster [DBG] pgmap v913: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:19 smithi156 bash[29655]: cluster 2023-11-17T06:02:17.953590+0000 mgr.smithi006.odexej (mgr.14176) 1333 : cluster [DBG] pgmap v914: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:21 smithi156 bash[29655]: cluster 2023-11-17T06:02:19.955103+0000 mgr.smithi006.odexej (mgr.14176) 1334 : cluster [DBG] pgmap v915: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:22 smithi156 bash[29655]: audit 2023-11-17T06:02:21.592164+0000 mon.smithi006 (mon.0) 989 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:22 smithi156 bash[29655]: cluster 2023-11-17T06:02:21.956311+0000 mgr.smithi006.odexej (mgr.14176) 1335 : cluster [DBG] pgmap v916: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.971 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:22 smithi156 bash[29655]: audit 2023-11-17T06:02:22.467984+0000 mon.smithi006 (mon.0) 990 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: audit 2023-11-17T06:02:22.756826+0000 mon.smithi006 (mon.0) 991 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: audit 2023-11-17T06:02:22.758209+0000 mon.smithi006 (mon.0) 992 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: audit 2023-11-17T06:02:22.767190+0000 mon.smithi006 (mon.0) 993 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: cluster 2023-11-17T06:02:22.768075+0000 mgr.smithi006.odexej (mgr.14176) 1336 : cluster [DBG] pgmap v917: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 189 B/s rd, 0 op/s 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: audit 2023-11-17T06:02:22.774796+0000 mon.smithi006 (mon.0) 994 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: audit 2023-11-17T06:02:22.790849+0000 mon.smithi006 (mon.0) 995 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:23 smithi156 bash[29655]: cephadm 2023-11-17T06:02:22.794877+0000 mgr.smithi006.odexej (mgr.14176) 1337 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.mvusrd on smithi156 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:24 smithi156 bash[29655]: cluster 2023-11-17T06:02:23.763075+0000 mon.smithi006 (mon.0) 996 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:24 smithi156 bash[29655]: cluster 2023-11-17T06:02:23.763178+0000 mon.smithi006 (mon.0) 997 : cluster [INF] Cluster is now healthy 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:25 smithi156 bash[29655]: cluster 2023-11-17T06:02:24.769174+0000 mgr.smithi006.odexej (mgr.14176) 1338 : cluster [DBG] pgmap v918: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:26 smithi156 bash[29655]: audit 2023-11-17T06:02:26.551492+0000 mon.smithi006 (mon.0) 998 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.mvusrd"}]: dispatch 2023-11-17T06:03:07.972 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: cephadm 2023-11-17T06:02:26.550117+0000 mgr.smithi006.odexej (mgr.14176) 1339 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.973 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.974 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: cephadm 2023-11-17T06:02:26.550692+0000 mgr.smithi006.odexej (mgr.14176) 1340 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: cephadm 2023-11-17T06:02:26.553620+0000 mgr.smithi006.odexej (mgr.14176) 1341 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.975 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: cephadm 2023-11-17T06:02:26.558834+0000 mgr.smithi006.odexej (mgr.14176) 1342 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.xsqjtn on smithi006 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:27 smithi156 bash[29655]: cluster 2023-11-17T06:02:26.769888+0000 mgr.smithi006.odexej (mgr.14176) 1343 : cluster [DBG] pgmap v919: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:29 smithi156 bash[29655]: cluster 2023-11-17T06:02:28.770800+0000 mgr.smithi006.odexej (mgr.14176) 1344 : cluster [DBG] pgmap v920: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:30 smithi156 bash[29655]: audit 2023-11-17T06:02:30.323554+0000 mon.smithi006 (mon.0) 999 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.xsqjtn"}]: dispatch 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cephadm 2023-11-17T06:02:30.322734+0000 mgr.smithi006.odexej (mgr.14176) 1345 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.976 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.977 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cephadm 2023-11-17T06:02:30.322966+0000 mgr.smithi006.odexej (mgr.14176) 1346 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cephadm 2023-11-17T06:02:30.324818+0000 mgr.smithi006.odexej (mgr.14176) 1347 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.978 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cephadm 2023-11-17T06:02:30.326727+0000 mgr.smithi006.odexej (mgr.14176) 1348 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cephadm 2023-11-17T06:02:30.328307+0000 mgr.smithi006.odexej (mgr.14176) 1349 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cluster 2023-11-17T06:02:30.329389+0000 mgr.smithi006.odexej (mgr.14176) 1350 : cluster [DBG] pgmap v921: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: cluster 2023-11-17T06:02:30.828434+0000 mon.smithi006 (mon.0) 1000 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:31 smithi156 bash[29655]: audit 2023-11-17T06:02:31.040294+0000 mon.smithi006 (mon.0) 1001 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.979 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:33 smithi156 bash[29655]: cluster 2023-11-17T06:02:32.330381+0000 mgr.smithi006.odexej (mgr.14176) 1351 : cluster [DBG] pgmap v922: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:35 smithi156 bash[29655]: cluster 2023-11-17T06:02:34.331702+0000 mgr.smithi006.odexej (mgr.14176) 1352 : cluster [DBG] pgmap v923: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:37 smithi156 bash[29655]: cluster 2023-11-17T06:02:36.332610+0000 mgr.smithi006.odexej (mgr.14176) 1353 : cluster [DBG] pgmap v924: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:39 smithi156 bash[29655]: cluster 2023-11-17T06:02:38.333601+0000 mgr.smithi006.odexej (mgr.14176) 1354 : cluster [DBG] pgmap v925: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:41 smithi156 bash[29655]: cluster 2023-11-17T06:02:40.334976+0000 mgr.smithi006.odexej (mgr.14176) 1355 : cluster [DBG] pgmap v926: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:43 smithi156 bash[29655]: cluster 2023-11-17T06:02:42.336069+0000 mgr.smithi006.odexej (mgr.14176) 1356 : cluster [DBG] pgmap v927: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:45 smithi156 bash[29655]: cluster 2023-11-17T06:02:44.337342+0000 mgr.smithi006.odexej (mgr.14176) 1357 : cluster [DBG] pgmap v928: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:47 smithi156 bash[29655]: cluster 2023-11-17T06:02:46.338629+0000 mgr.smithi006.odexej (mgr.14176) 1358 : cluster [DBG] pgmap v929: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:49 smithi156 bash[29655]: cluster 2023-11-17T06:02:48.339732+0000 mgr.smithi006.odexej (mgr.14176) 1359 : cluster [DBG] pgmap v930: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:51 smithi156 bash[29655]: cluster 2023-11-17T06:02:50.341447+0000 mgr.smithi006.odexej (mgr.14176) 1360 : cluster [DBG] pgmap v931: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:53 smithi156 bash[29655]: cluster 2023-11-17T06:02:52.342391+0000 mgr.smithi006.odexej (mgr.14176) 1361 : cluster [DBG] pgmap v932: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:55 smithi156 bash[29655]: cluster 2023-11-17T06:02:54.343841+0000 mgr.smithi006.odexej (mgr.14176) 1362 : cluster [DBG] pgmap v933: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:57 smithi156 bash[29655]: cluster 2023-11-17T06:02:56.344909+0000 mgr.smithi006.odexej (mgr.14176) 1363 : cluster [DBG] pgmap v934: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:02:59 smithi156 bash[29655]: cluster 2023-11-17T06:02:58.345976+0000 mgr.smithi006.odexej (mgr.14176) 1364 : cluster [DBG] pgmap v935: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.980 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:01 smithi156 bash[29655]: cluster 2023-11-17T06:03:00.347466+0000 mgr.smithi006.odexej (mgr.14176) 1365 : cluster [DBG] pgmap v936: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:03 smithi156 bash[29655]: cluster 2023-11-17T06:03:02.348454+0000 mgr.smithi006.odexej (mgr.14176) 1366 : cluster [DBG] pgmap v937: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:05 smithi156 bash[29655]: cluster 2023-11-17T06:03:04.349867+0000 mgr.smithi006.odexej (mgr.14176) 1367 : cluster [DBG] pgmap v938: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:57 smithi006 bash[26022]: audit 2023-11-17T06:01:55.329019+0000 mgr.smithi006.odexej (mgr.14176) 1321 : audit [DBG] from='client.15218 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:57 smithi006 bash[26022]: cluster 2023-11-17T06:01:55.940504+0000 mgr.smithi006.odexej (mgr.14176) 1322 : cluster [DBG] pgmap v903: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:01:59 smithi006 bash[26022]: cluster 2023-11-17T06:01:57.941494+0000 mgr.smithi006.odexej (mgr.14176) 1323 : cluster [DBG] pgmap v904: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:01 smithi006 bash[26022]: cluster 2023-11-17T06:01:59.942850+0000 mgr.smithi006.odexej (mgr.14176) 1324 : cluster [DBG] pgmap v905: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:02 smithi006 bash[26022]: cluster 2023-11-17T06:02:01.943887+0000 mgr.smithi006.odexej (mgr.14176) 1325 : cluster [DBG] pgmap v906: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:05 smithi006 bash[26022]: cluster 2023-11-17T06:02:03.945053+0000 mgr.smithi006.odexej (mgr.14176) 1326 : cluster [DBG] pgmap v907: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:07 smithi006 bash[26022]: cluster 2023-11-17T06:02:05.946385+0000 mgr.smithi006.odexej (mgr.14176) 1327 : cluster [DBG] pgmap v908: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:09 smithi006 bash[26022]: cluster 2023-11-17T06:02:07.947448+0000 mgr.smithi006.odexej (mgr.14176) 1328 : cluster [DBG] pgmap v909: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:11 smithi006 bash[26022]: cluster 2023-11-17T06:02:09.948899+0000 mgr.smithi006.odexej (mgr.14176) 1329 : cluster [DBG] pgmap v910: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:12 smithi006 bash[26022]: cluster 2023-11-17T06:02:11.950111+0000 mgr.smithi006.odexej (mgr.14176) 1330 : cluster [DBG] pgmap v911: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.981 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:15 smithi006 bash[26022]: cluster 2023-11-17T06:02:13.951271+0000 mgr.smithi006.odexej (mgr.14176) 1331 : cluster [DBG] pgmap v912: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:16 smithi006 bash[26022]: audit 2023-11-17T06:02:15.919172+0000 mon.smithi006 (mon.0) 988 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:17 smithi006 bash[26022]: cluster 2023-11-17T06:02:15.952431+0000 mgr.smithi006.odexej (mgr.14176) 1332 : cluster [DBG] pgmap v913: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:19 smithi006 bash[26022]: cluster 2023-11-17T06:02:17.953590+0000 mgr.smithi006.odexej (mgr.14176) 1333 : cluster [DBG] pgmap v914: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:21 smithi006 bash[26022]: cluster 2023-11-17T06:02:19.955103+0000 mgr.smithi006.odexej (mgr.14176) 1334 : cluster [DBG] pgmap v915: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:22 smithi006 bash[26022]: audit 2023-11-17T06:02:21.592164+0000 mon.smithi006 (mon.0) 989 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:22 smithi006 bash[26022]: cluster 2023-11-17T06:02:21.956311+0000 mgr.smithi006.odexej (mgr.14176) 1335 : cluster [DBG] pgmap v916: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:22 smithi006 bash[26022]: audit 2023-11-17T06:02:22.467984+0000 mon.smithi006 (mon.0) 990 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: audit 2023-11-17T06:02:22.756826+0000 mon.smithi006 (mon.0) 991 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: audit 2023-11-17T06:02:22.758209+0000 mon.smithi006 (mon.0) 992 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: audit 2023-11-17T06:02:22.767190+0000 mon.smithi006 (mon.0) 993 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: cluster 2023-11-17T06:02:22.768075+0000 mgr.smithi006.odexej (mgr.14176) 1336 : cluster [DBG] pgmap v917: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 189 B/s rd, 0 op/s 2023-11-17T06:03:07.982 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: audit 2023-11-17T06:02:22.774796+0000 mon.smithi006 (mon.0) 994 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: audit 2023-11-17T06:02:22.790849+0000 mon.smithi006 (mon.0) 995 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:23 smithi006 bash[26022]: cephadm 2023-11-17T06:02:22.794877+0000 mgr.smithi006.odexej (mgr.14176) 1337 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.mvusrd on smithi156 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:24 smithi006 bash[26022]: cluster 2023-11-17T06:02:23.763075+0000 mon.smithi006 (mon.0) 996 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:24 smithi006 bash[26022]: cluster 2023-11-17T06:02:23.763178+0000 mon.smithi006 (mon.0) 997 : cluster [INF] Cluster is now healthy 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:25 smithi006 bash[26022]: cluster 2023-11-17T06:02:24.769174+0000 mgr.smithi006.odexej (mgr.14176) 1338 : cluster [DBG] pgmap v918: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:26 smithi006 bash[26022]: audit 2023-11-17T06:02:26.551492+0000 mon.smithi006 (mon.0) 998 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.mvusrd"}]: dispatch 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: cephadm 2023-11-17T06:02:26.550117+0000 mgr.smithi006.odexej (mgr.14176) 1339 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.983 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.984 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: cephadm 2023-11-17T06:02:26.550692+0000 mgr.smithi006.odexej (mgr.14176) 1340 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: cephadm 2023-11-17T06:02:26.553620+0000 mgr.smithi006.odexej (mgr.14176) 1341 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.985 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.mvusrd ... 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: cephadm 2023-11-17T06:02:26.558834+0000 mgr.smithi006.odexej (mgr.14176) 1342 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.xsqjtn on smithi006 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:27 smithi006 bash[26022]: cluster 2023-11-17T06:02:26.769888+0000 mgr.smithi006.odexej (mgr.14176) 1343 : cluster [DBG] pgmap v919: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:29 smithi006 bash[26022]: cluster 2023-11-17T06:02:28.770800+0000 mgr.smithi006.odexej (mgr.14176) 1344 : cluster [DBG] pgmap v920: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 94 B/s rd, 0 op/s 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:30 smithi006 bash[26022]: audit 2023-11-17T06:02:30.323554+0000 mon.smithi006 (mon.0) 999 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.xsqjtn"}]: dispatch 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cephadm 2023-11-17T06:02:30.322734+0000 mgr.smithi006.odexej (mgr.14176) 1345 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.986 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:07.987 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.988 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cephadm 2023-11-17T06:02:30.322966+0000 mgr.smithi006.odexej (mgr.14176) 1346 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cephadm 2023-11-17T06:02:30.324818+0000 mgr.smithi006.odexej (mgr.14176) 1347 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.xsqjtn ... 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.989 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cephadm 2023-11-17T06:02:30.326727+0000 mgr.smithi006.odexej (mgr.14176) 1348 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cephadm 2023-11-17T06:02:30.328307+0000 mgr.smithi006.odexej (mgr.14176) 1349 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cluster 2023-11-17T06:02:30.329389+0000 mgr.smithi006.odexej (mgr.14176) 1350 : cluster [DBG] pgmap v921: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: cluster 2023-11-17T06:02:30.828434+0000 mon.smithi006 (mon.0) 1000 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:31 smithi006 bash[26022]: audit 2023-11-17T06:02:31.040294+0000 mon.smithi006 (mon.0) 1001 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:33 smithi006 bash[26022]: cluster 2023-11-17T06:02:32.330381+0000 mgr.smithi006.odexej (mgr.14176) 1351 : cluster [DBG] pgmap v922: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:35 smithi006 bash[26022]: cluster 2023-11-17T06:02:34.331702+0000 mgr.smithi006.odexej (mgr.14176) 1352 : cluster [DBG] pgmap v923: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:37 smithi006 bash[26022]: cluster 2023-11-17T06:02:36.332610+0000 mgr.smithi006.odexej (mgr.14176) 1353 : cluster [DBG] pgmap v924: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:39 smithi006 bash[26022]: cluster 2023-11-17T06:02:38.333601+0000 mgr.smithi006.odexej (mgr.14176) 1354 : cluster [DBG] pgmap v925: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.990 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:41 smithi006 bash[26022]: cluster 2023-11-17T06:02:40.334976+0000 mgr.smithi006.odexej (mgr.14176) 1355 : cluster [DBG] pgmap v926: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:43 smithi006 bash[26022]: cluster 2023-11-17T06:02:42.336069+0000 mgr.smithi006.odexej (mgr.14176) 1356 : cluster [DBG] pgmap v927: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:45 smithi006 bash[26022]: cluster 2023-11-17T06:02:44.337342+0000 mgr.smithi006.odexej (mgr.14176) 1357 : cluster [DBG] pgmap v928: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:47 smithi006 bash[26022]: cluster 2023-11-17T06:02:46.338629+0000 mgr.smithi006.odexej (mgr.14176) 1358 : cluster [DBG] pgmap v929: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:49 smithi006 bash[26022]: cluster 2023-11-17T06:02:48.339732+0000 mgr.smithi006.odexej (mgr.14176) 1359 : cluster [DBG] pgmap v930: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:51 smithi006 bash[26022]: cluster 2023-11-17T06:02:50.341447+0000 mgr.smithi006.odexej (mgr.14176) 1360 : cluster [DBG] pgmap v931: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:53 smithi006 bash[26022]: cluster 2023-11-17T06:02:52.342391+0000 mgr.smithi006.odexej (mgr.14176) 1361 : cluster [DBG] pgmap v932: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:55 smithi006 bash[26022]: cluster 2023-11-17T06:02:54.343841+0000 mgr.smithi006.odexej (mgr.14176) 1362 : cluster [DBG] pgmap v933: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:57 smithi006 bash[26022]: cluster 2023-11-17T06:02:56.344909+0000 mgr.smithi006.odexej (mgr.14176) 1363 : cluster [DBG] pgmap v934: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:02:59 smithi006 bash[26022]: cluster 2023-11-17T06:02:58.345976+0000 mgr.smithi006.odexej (mgr.14176) 1364 : cluster [DBG] pgmap v935: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:01 smithi006 bash[26022]: cluster 2023-11-17T06:03:00.347466+0000 mgr.smithi006.odexej (mgr.14176) 1365 : cluster [DBG] pgmap v936: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:03 smithi006 bash[26022]: cluster 2023-11-17T06:03:02.348454+0000 mgr.smithi006.odexej (mgr.14176) 1366 : cluster [DBG] pgmap v937: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:07.991 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:05 smithi006 bash[26022]: cluster 2023-11-17T06:03:04.349867+0000 mgr.smithi006.odexej (mgr.14176) 1367 : cluster [DBG] pgmap v938: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:08.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:07 smithi006 bash[26022]: cluster 2023-11-17T06:03:06.350981+0000 mgr.smithi006.odexej (mgr.14176) 1368 : cluster [DBG] pgmap v939: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:08.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:07 smithi156 bash[29655]: cluster 2023-11-17T06:03:06.350981+0000 mgr.smithi006.odexej (mgr.14176) 1368 : cluster [DBG] pgmap v939: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:10.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:09 smithi006 bash[26022]: cluster 2023-11-17T06:03:08.352248+0000 mgr.smithi006.odexej (mgr.14176) 1369 : cluster [DBG] pgmap v940: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:10.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:09 smithi156 bash[29655]: cluster 2023-11-17T06:03:08.352248+0000 mgr.smithi006.odexej (mgr.14176) 1369 : cluster [DBG] pgmap v940: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:11.652 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:12.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:11 smithi006 bash[26022]: cluster 2023-11-17T06:03:10.353743+0000 mgr.smithi006.odexej (mgr.14176) 1370 : cluster [DBG] pgmap v941: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:11 smithi006 bash[26022]: audit 2023-11-17T06:03:11.077661+0000 mgr.smithi006.odexej (mgr.14176) 1371 : audit [DBG] from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:12.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:11 smithi156 bash[29655]: cluster 2023-11-17T06:03:10.353743+0000 mgr.smithi006.odexej (mgr.14176) 1370 : cluster [DBG] pgmap v941: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:12.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:11 smithi156 bash[29655]: audit 2023-11-17T06:03:11.077661+0000 mgr.smithi006.odexej (mgr.14176) 1371 : audit [DBG] from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:12.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:14.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:13 smithi006 bash[26022]: cluster 2023-11-17T06:03:12.354692+0000 mgr.smithi006.odexej (mgr.14176) 1372 : cluster [DBG] pgmap v942: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:14.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:13 smithi156 bash[29655]: cluster 2023-11-17T06:03:12.354692+0000 mgr.smithi006.odexej (mgr.14176) 1372 : cluster [DBG] pgmap v942: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:15.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:15.658 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:16.257 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:15 smithi006 bash[26022]: cluster 2023-11-17T06:03:14.356132+0000 mgr.smithi006.odexej (mgr.14176) 1373 : cluster [DBG] pgmap v943: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:16.258 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:16.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:15 smithi156 bash[29655]: cluster 2023-11-17T06:03:14.356132+0000 mgr.smithi006.odexej (mgr.14176) 1373 : cluster [DBG] pgmap v943: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:17.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:17.277 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:16 smithi006 bash[26022]: audit 2023-11-17T06:03:15.641116+0000 mgr.smithi006.odexej (mgr.14176) 1374 : audit [DBG] from='client.15226 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:17.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:16 smithi156 bash[29655]: audit 2023-11-17T06:03:15.641116+0000 mgr.smithi006.odexej (mgr.14176) 1374 : audit [DBG] from='client.15226 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:18.271 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:17 smithi006 bash[26022]: cluster 2023-11-17T06:03:16.357205+0000 mgr.smithi006.odexej (mgr.14176) 1375 : cluster [DBG] pgmap v944: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:18.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:17 smithi156 bash[29655]: cluster 2023-11-17T06:03:16.357205+0000 mgr.smithi006.odexej (mgr.14176) 1375 : cluster [DBG] pgmap v944: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:20.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:20 smithi006 bash[26022]: cluster 2023-11-17T06:03:18.358400+0000 mgr.smithi006.odexej (mgr.14176) 1376 : cluster [DBG] pgmap v945: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:20.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:20.260 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:20.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:20 smithi156 bash[29655]: cluster 2023-11-17T06:03:18.358400+0000 mgr.smithi006.odexej (mgr.14176) 1376 : cluster [DBG] pgmap v945: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:20.913 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:21.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:22.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:22 smithi006 bash[26022]: audit 2023-11-17T06:03:20.244881+0000 mgr.smithi006.odexej (mgr.14176) 1377 : audit [DBG] from='client.15230 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:22.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:22 smithi006 bash[26022]: cluster 2023-11-17T06:03:20.359998+0000 mgr.smithi006.odexej (mgr.14176) 1378 : cluster [DBG] pgmap v946: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:22.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:22 smithi156 bash[29655]: audit 2023-11-17T06:03:20.244881+0000 mgr.smithi006.odexej (mgr.14176) 1377 : audit [DBG] from='client.15230 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:22.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:22 smithi156 bash[29655]: cluster 2023-11-17T06:03:20.359998+0000 mgr.smithi006.odexej (mgr.14176) 1378 : cluster [DBG] pgmap v946: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:24.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:24 smithi006 bash[26022]: cluster 2023-11-17T06:03:22.361021+0000 mgr.smithi006.odexej (mgr.14176) 1379 : cluster [DBG] pgmap v947: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:24.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:24 smithi156 bash[29655]: cluster 2023-11-17T06:03:22.361021+0000 mgr.smithi006.odexej (mgr.14176) 1379 : cluster [DBG] pgmap v947: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:25.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:25.187 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:25.807 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:26.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:26 smithi006 bash[26022]: cluster 2023-11-17T06:03:24.362342+0000 mgr.smithi006.odexej (mgr.14176) 1380 : cluster [DBG] pgmap v948: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:26.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:26 smithi006 bash[26022]: audit 2023-11-17T06:03:25.169954+0000 mgr.smithi006.odexej (mgr.14176) 1381 : audit [DBG] from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:26.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:26 smithi156 bash[29655]: cluster 2023-11-17T06:03:24.362342+0000 mgr.smithi006.odexej (mgr.14176) 1380 : cluster [DBG] pgmap v948: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:03:26.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:26 smithi156 bash[29655]: audit 2023-11-17T06:03:25.169954+0000 mgr.smithi006.odexej (mgr.14176) 1381 : audit [DBG] from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:26.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:28.070 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:28 smithi006 bash[26022]: cluster 2023-11-17T06:03:26.363443+0000 mgr.smithi006.odexej (mgr.14176) 1382 : cluster [DBG] pgmap v949: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:28.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:28 smithi156 bash[29655]: cluster 2023-11-17T06:03:26.363443+0000 mgr.smithi006.odexej (mgr.14176) 1382 : cluster [DBG] pgmap v949: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:29.753 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:30.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:30 smithi006 bash[26022]: cluster 2023-11-17T06:03:28.364627+0000 mgr.smithi006.odexej (mgr.14176) 1383 : cluster [DBG] pgmap v950: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:30.322 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:30 smithi156 bash[29655]: cluster 2023-11-17T06:03:28.364627+0000 mgr.smithi006.odexej (mgr.14176) 1383 : cluster [DBG] pgmap v950: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:31.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:29.744021+0000 mgr.smithi006.odexej (mgr.14176) 1384 : audit [DBG] from='client.15238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.332231+0000 mon.smithi006 (mon.0) 1002 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.640006+0000 mon.smithi006 (mon.0) 1003 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.964197+0000 mon.smithi006 (mon.0) 1004 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.965636+0000 mon.smithi006 (mon.0) 1005 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.973385+0000 mon.smithi006 (mon.0) 1006 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.981740+0000 mon.smithi006 (mon.0) 1007 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:03:31.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:31 smithi006 bash[26022]: audit 2023-11-17T06:03:30.997009+0000 mon.smithi006 (mon.0) 1008 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:29.744021+0000 mgr.smithi006.odexej (mgr.14176) 1384 : audit [DBG] from='client.15238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.332231+0000 mon.smithi006 (mon.0) 1002 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.640006+0000 mon.smithi006 (mon.0) 1003 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.964197+0000 mon.smithi006 (mon.0) 1004 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.965636+0000 mon.smithi006 (mon.0) 1005 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.973385+0000 mon.smithi006 (mon.0) 1006 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.981740+0000 mon.smithi006 (mon.0) 1007 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:03:31.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:31 smithi156 bash[29655]: audit 2023-11-17T06:03:30.997009+0000 mon.smithi006 (mon.0) 1008 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:32.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:32 smithi006 bash[26022]: cluster 2023-11-17T06:03:30.365951+0000 mgr.smithi006.odexej (mgr.14176) 1385 : cluster [DBG] pgmap v951: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:32.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:32 smithi006 bash[26022]: cluster 2023-11-17T06:03:30.974172+0000 mgr.smithi006.odexej (mgr.14176) 1386 : cluster [DBG] pgmap v952: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:32.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:32 smithi006 bash[26022]: cephadm 2023-11-17T06:03:31.001143+0000 mgr.smithi006.odexej (mgr.14176) 1387 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dygrfo on smithi156 2023-11-17T06:03:32.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:32 smithi006 bash[26022]: cluster 2023-11-17T06:03:31.029387+0000 mon.smithi006 (mon.0) 1009 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:03:32.361 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:32 smithi006 bash[26022]: cluster 2023-11-17T06:03:31.029449+0000 mon.smithi006 (mon.0) 1010 : cluster [INF] Cluster is now healthy 2023-11-17T06:03:32.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:32 smithi156 bash[29655]: cluster 2023-11-17T06:03:30.365951+0000 mgr.smithi006.odexej (mgr.14176) 1385 : cluster [DBG] pgmap v951: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:03:32.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:32 smithi156 bash[29655]: cluster 2023-11-17T06:03:30.974172+0000 mgr.smithi006.odexej (mgr.14176) 1386 : cluster [DBG] pgmap v952: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:32.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:32 smithi156 bash[29655]: cephadm 2023-11-17T06:03:31.001143+0000 mgr.smithi006.odexej (mgr.14176) 1387 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dygrfo on smithi156 2023-11-17T06:03:32.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:32 smithi156 bash[29655]: cluster 2023-11-17T06:03:31.029387+0000 mon.smithi006 (mon.0) 1009 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:03:32.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:32 smithi156 bash[29655]: cluster 2023-11-17T06:03:31.029449+0000 mon.smithi006 (mon.0) 1010 : cluster [INF] Cluster is now healthy 2023-11-17T06:03:34.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:34 smithi006 bash[26022]: cluster 2023-11-17T06:03:32.974974+0000 mgr.smithi006.odexej (mgr.14176) 1388 : cluster [DBG] pgmap v953: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T06:03:34.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:34.279 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:01:15.914033Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.iaveoh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:26.553356Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.mvusrd on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-mvusrd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.mvusrd\nDeploy daemon haproxy.nfs.foo.smithi156.mvusrd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.324690Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.xsqjtn on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-xsqjtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.xsqjtn\nDeploy daemon haproxy.nfs.foo.smithi006.xsqjtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:02:30.326623Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rnxmyz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:34.407 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:34 smithi156 bash[29655]: cluster 2023-11-17T06:03:32.974974+0000 mgr.smithi006.odexej (mgr.14176) 1388 : cluster [DBG] pgmap v953: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T06:03:34.913 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:03:35.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:03:36.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: audit 2023-11-17T06:03:34.267448+0000 mgr.smithi006.odexej (mgr.14176) 1389 : audit [DBG] from='client.15242 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:36.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: cephadm 2023-11-17T06:03:34.735791+0000 mgr.smithi006.odexej (mgr.14176) 1390 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: cephadm 2023-11-17T06:03:34.736343+0000 mgr.smithi006.odexej (mgr.14176) 1391 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: audit 2023-11-17T06:03:34.738932+0000 mon.smithi006 (mon.0) 1011 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:36.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: audit 2023-11-17T06:03:34.739393+0000 mon.smithi006 (mon.0) 1012 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dygrfo"}]: dispatch 2023-11-17T06:03:36.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: cephadm 2023-11-17T06:03:34.741633+0000 mgr.smithi006.odexej (mgr.14176) 1392 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:03:36.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.171 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: cephadm 2023-11-17T06:03:34.745602+0000 mgr.smithi006.odexej (mgr.14176) 1393 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.mrqief on smithi006 2023-11-17T06:03:36.173 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:35 smithi006 bash[26022]: cluster 2023-11-17T06:03:34.976179+0000 mgr.smithi006.odexej (mgr.14176) 1394 : cluster [DBG] pgmap v954: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:36.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: audit 2023-11-17T06:03:34.267448+0000 mgr.smithi006.odexej (mgr.14176) 1389 : audit [DBG] from='client.15242 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:03:36.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: cephadm 2023-11-17T06:03:34.735791+0000 mgr.smithi006.odexej (mgr.14176) 1390 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.189 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.194 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.194 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: cephadm 2023-11-17T06:03:34.736343+0000 mgr.smithi006.odexej (mgr.14176) 1391 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: audit 2023-11-17T06:03:34.738932+0000 mon.smithi006 (mon.0) 1011 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: audit 2023-11-17T06:03:34.739393+0000 mon.smithi006 (mon.0) 1012 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dygrfo"}]: dispatch 2023-11-17T06:03:36.195 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: cephadm 2023-11-17T06:03:34.741633+0000 mgr.smithi006.odexej (mgr.14176) 1392 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dygrfo ... 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: cephadm 2023-11-17T06:03:34.745602+0000 mgr.smithi006.odexej (mgr.14176) 1393 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.mrqief on smithi006 2023-11-17T06:03:36.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:35 smithi156 bash[29655]: cluster 2023-11-17T06:03:34.976179+0000 mgr.smithi006.odexej (mgr.14176) 1394 : cluster [DBG] pgmap v954: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:38.150 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:37 smithi006 bash[26022]: cluster 2023-11-17T06:03:36.977027+0000 mgr.smithi006.odexej (mgr.14176) 1395 : cluster [DBG] pgmap v955: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:38.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:37 smithi156 bash[29655]: cluster 2023-11-17T06:03:36.977027+0000 mgr.smithi006.odexej (mgr.14176) 1395 : cluster [DBG] pgmap v955: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:03:39.065 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:03:39.065 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:03:39.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:38 smithi006 bash[26022]: audit 2023-11-17T06:03:38.587544+0000 mon.smithi006 (mon.0) 1013 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.mrqief"}]: dispatch 2023-11-17T06:03:39.188 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:38 smithi156 bash[29655]: audit 2023-11-17T06:03:38.587544+0000 mon.smithi006 (mon.0) 1013 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.mrqief"}]: dispatch 2023-11-17T06:03:39.745 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:14.123 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cephadm 2023-11-17T06:03:38.586293+0000 mgr.smithi006.odexej (mgr.14176) 1396 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.161 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.239 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.243 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:14.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:14.446 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:14.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:14.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:04:14.447 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:14.692 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:04:14.693 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:14.693 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:04:14.693 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.693 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.693 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.718 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.718 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:14.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:14.720 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.720 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cephadm 2023-11-17T06:03:38.586797+0000 mgr.smithi006.odexej (mgr.14176) 1397 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.720 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cephadm 2023-11-17T06:03:38.589130+0000 mgr.smithi006.odexej (mgr.14176) 1398 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.720 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.720 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:14.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:14.721 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.722 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:14.722 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:14.725 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:14.725 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:14.725 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cephadm 2023-11-17T06:03:38.593567+0000 mgr.smithi006.odexej (mgr.14176) 1399 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:14.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cephadm 2023-11-17T06:03:38.595598+0000 mgr.smithi006.odexej (mgr.14176) 1400 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:14.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cluster 2023-11-17T06:03:38.596982+0000 mgr.smithi006.odexej (mgr.14176) 1401 : cluster [DBG] pgmap v956: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:04:14.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: cluster 2023-11-17T06:03:38.759585+0000 mon.smithi006 (mon.0) 1014 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:04:14.753 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:39 smithi156 bash[29655]: audit 2023-11-17T06:03:39.049772+0000 mgr.smithi006.odexej (mgr.14176) 1402 : audit [DBG] from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:14.754 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:42 smithi156 bash[29655]: cluster 2023-11-17T06:03:40.598371+0000 mgr.smithi006.odexej (mgr.14176) 1403 : cluster [DBG] pgmap v957: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:04:14.754 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:42 smithi156 bash[29655]: audit 2023-11-17T06:03:41.095303+0000 mon.smithi006 (mon.0) 1015 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:14.754 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:44 smithi156 bash[29655]: cluster 2023-11-17T06:03:42.599432+0000 mgr.smithi006.odexej (mgr.14176) 1404 : cluster [DBG] pgmap v958: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T06:04:14.754 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:46 smithi156 bash[29655]: cluster 2023-11-17T06:03:44.600832+0000 mgr.smithi006.odexej (mgr.14176) 1405 : cluster [DBG] pgmap v959: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:14.754 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:48 smithi156 bash[29655]: cluster 2023-11-17T06:03:46.601896+0000 mgr.smithi006.odexej (mgr.14176) 1406 : cluster [DBG] pgmap v960: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:50 smithi156 bash[29655]: cluster 2023-11-17T06:03:48.603095+0000 mgr.smithi006.odexej (mgr.14176) 1407 : cluster [DBG] pgmap v961: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:50 smithi156 bash[29655]: audit 2023-11-17T06:03:49.738734+0000 mon.smithi006 (mon.0) 1016 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:52 smithi156 bash[29655]: cluster 2023-11-17T06:03:50.604491+0000 mgr.smithi006.odexej (mgr.14176) 1408 : cluster [DBG] pgmap v962: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:54 smithi156 bash[29655]: cluster 2023-11-17T06:03:52.605542+0000 mgr.smithi006.odexej (mgr.14176) 1409 : cluster [DBG] pgmap v963: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:56 smithi156 bash[29655]: cluster 2023-11-17T06:03:54.606962+0000 mgr.smithi006.odexej (mgr.14176) 1410 : cluster [DBG] pgmap v964: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.755 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:03:58 smithi156 bash[29655]: cluster 2023-11-17T06:03:56.608073+0000 mgr.smithi006.odexej (mgr.14176) 1411 : cluster [DBG] pgmap v965: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:00 smithi156 bash[29655]: cluster 2023-11-17T06:03:58.609224+0000 mgr.smithi006.odexej (mgr.14176) 1412 : cluster [DBG] pgmap v966: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:02 smithi156 bash[29655]: cluster 2023-11-17T06:04:00.610600+0000 mgr.smithi006.odexej (mgr.14176) 1413 : cluster [DBG] pgmap v967: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:04 smithi156 bash[29655]: cluster 2023-11-17T06:04:02.611637+0000 mgr.smithi006.odexej (mgr.14176) 1414 : cluster [DBG] pgmap v968: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:06 smithi156 bash[29655]: cluster 2023-11-17T06:04:04.613006+0000 mgr.smithi006.odexej (mgr.14176) 1415 : cluster [DBG] pgmap v969: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:08 smithi156 bash[29655]: cluster 2023-11-17T06:04:06.614038+0000 mgr.smithi006.odexej (mgr.14176) 1416 : cluster [DBG] pgmap v970: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:10 smithi156 bash[29655]: cluster 2023-11-17T06:04:08.615183+0000 mgr.smithi006.odexej (mgr.14176) 1417 : cluster [DBG] pgmap v971: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.757 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:12 smithi156 bash[29655]: cluster 2023-11-17T06:04:10.616728+0000 mgr.smithi006.odexej (mgr.14176) 1418 : cluster [DBG] pgmap v972: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:14.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cephadm 2023-11-17T06:03:38.586293+0000 mgr.smithi006.odexej (mgr.14176) 1396 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:14.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:14.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:14.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:15.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:15.015 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cephadm 2023-11-17T06:03:38.586797+0000 mgr.smithi006.odexej (mgr.14176) 1397 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cephadm 2023-11-17T06:03:38.589130+0000 mgr.smithi006.odexej (mgr.14176) 1398 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:15.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.mrqief ... 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cephadm 2023-11-17T06:03:38.593567+0000 mgr.smithi006.odexej (mgr.14176) 1399 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cephadm 2023-11-17T06:03:38.595598+0000 mgr.smithi006.odexej (mgr.14176) 1400 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cluster 2023-11-17T06:03:38.596982+0000 mgr.smithi006.odexej (mgr.14176) 1401 : cluster [DBG] pgmap v956: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: cluster 2023-11-17T06:03:38.759585+0000 mon.smithi006 (mon.0) 1014 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:39 smithi006 bash[26022]: audit 2023-11-17T06:03:39.049772+0000 mgr.smithi006.odexej (mgr.14176) 1402 : audit [DBG] from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:15.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:42 smithi006 bash[26022]: cluster 2023-11-17T06:03:40.598371+0000 mgr.smithi006.odexej (mgr.14176) 1403 : cluster [DBG] pgmap v957: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:42 smithi006 bash[26022]: audit 2023-11-17T06:03:41.095303+0000 mon.smithi006 (mon.0) 1015 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:44 smithi006 bash[26022]: cluster 2023-11-17T06:03:42.599432+0000 mgr.smithi006.odexej (mgr.14176) 1404 : cluster [DBG] pgmap v958: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:46 smithi006 bash[26022]: cluster 2023-11-17T06:03:44.600832+0000 mgr.smithi006.odexej (mgr.14176) 1405 : cluster [DBG] pgmap v959: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:48 smithi006 bash[26022]: cluster 2023-11-17T06:03:46.601896+0000 mgr.smithi006.odexej (mgr.14176) 1406 : cluster [DBG] pgmap v960: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:50 smithi006 bash[26022]: cluster 2023-11-17T06:03:48.603095+0000 mgr.smithi006.odexej (mgr.14176) 1407 : cluster [DBG] pgmap v961: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:50 smithi006 bash[26022]: audit 2023-11-17T06:03:49.738734+0000 mon.smithi006 (mon.0) 1016 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:52 smithi006 bash[26022]: cluster 2023-11-17T06:03:50.604491+0000 mgr.smithi006.odexej (mgr.14176) 1408 : cluster [DBG] pgmap v962: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:54 smithi006 bash[26022]: cluster 2023-11-17T06:03:52.605542+0000 mgr.smithi006.odexej (mgr.14176) 1409 : cluster [DBG] pgmap v963: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:56 smithi006 bash[26022]: cluster 2023-11-17T06:03:54.606962+0000 mgr.smithi006.odexej (mgr.14176) 1410 : cluster [DBG] pgmap v964: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:03:58 smithi006 bash[26022]: cluster 2023-11-17T06:03:56.608073+0000 mgr.smithi006.odexej (mgr.14176) 1411 : cluster [DBG] pgmap v965: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:00 smithi006 bash[26022]: cluster 2023-11-17T06:03:58.609224+0000 mgr.smithi006.odexej (mgr.14176) 1412 : cluster [DBG] pgmap v966: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:02 smithi006 bash[26022]: cluster 2023-11-17T06:04:00.610600+0000 mgr.smithi006.odexej (mgr.14176) 1413 : cluster [DBG] pgmap v967: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:04 smithi006 bash[26022]: cluster 2023-11-17T06:04:02.611637+0000 mgr.smithi006.odexej (mgr.14176) 1414 : cluster [DBG] pgmap v968: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:15.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:06 smithi006 bash[26022]: cluster 2023-11-17T06:04:04.613006+0000 mgr.smithi006.odexej (mgr.14176) 1415 : cluster [DBG] pgmap v969: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:08 smithi006 bash[26022]: cluster 2023-11-17T06:04:06.614038+0000 mgr.smithi006.odexej (mgr.14176) 1416 : cluster [DBG] pgmap v970: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:10 smithi006 bash[26022]: cluster 2023-11-17T06:04:08.615183+0000 mgr.smithi006.odexej (mgr.14176) 1417 : cluster [DBG] pgmap v971: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:12 smithi006 bash[26022]: cluster 2023-11-17T06:04:10.616728+0000 mgr.smithi006.odexej (mgr.14176) 1418 : cluster [DBG] pgmap v972: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:15.019 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:14 smithi156 bash[29655]: cluster 2023-11-17T06:04:12.617839+0000 mgr.smithi006.odexej (mgr.14176) 1419 : cluster [DBG] pgmap v973: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:15.020 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:14 smithi006 bash[26022]: cluster 2023-11-17T06:04:12.617839+0000 mgr.smithi006.odexej (mgr.14176) 1419 : cluster [DBG] pgmap v973: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:15.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:16.574 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:16 smithi006 bash[26022]: cluster 2023-11-17T06:04:14.619233+0000 mgr.smithi006.odexej (mgr.14176) 1420 : cluster [DBG] pgmap v974: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:16 smithi156 bash[29655]: cluster 2023-11-17T06:04:14.619233+0000 mgr.smithi006.odexej (mgr.14176) 1420 : cluster [DBG] pgmap v974: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:18.268 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:18.270 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:18.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:18 smithi006 bash[26022]: cluster 2023-11-17T06:04:16.620418+0000 mgr.smithi006.odexej (mgr.14176) 1421 : cluster [DBG] pgmap v975: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:18.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:18 smithi156 bash[29655]: cluster 2023-11-17T06:04:16.620418+0000 mgr.smithi006.odexej (mgr.14176) 1421 : cluster [DBG] pgmap v975: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:18.892 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:19.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:20 smithi006 bash[26022]: audit 2023-11-17T06:04:18.246793+0000 mgr.smithi006.odexej (mgr.14176) 1422 : audit [DBG] from='client.15250 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:20 smithi006 bash[26022]: cluster 2023-11-17T06:04:18.621598+0000 mgr.smithi006.odexej (mgr.14176) 1423 : cluster [DBG] pgmap v976: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:20 smithi156 bash[29655]: audit 2023-11-17T06:04:18.246793+0000 mgr.smithi006.odexej (mgr.14176) 1422 : audit [DBG] from='client.15250 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:20 smithi156 bash[29655]: cluster 2023-11-17T06:04:18.621598+0000 mgr.smithi006.odexej (mgr.14176) 1423 : cluster [DBG] pgmap v976: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:22.547 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:22 smithi006 bash[26022]: cluster 2023-11-17T06:04:20.622926+0000 mgr.smithi006.odexej (mgr.14176) 1424 : cluster [DBG] pgmap v977: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:22.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:22 smithi156 bash[29655]: cluster 2023-11-17T06:04:20.622926+0000 mgr.smithi006.odexej (mgr.14176) 1424 : cluster [DBG] pgmap v977: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:22.905 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:22.905 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:23.503 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:24.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:24.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:24 smithi006 bash[26022]: cluster 2023-11-17T06:04:22.624008+0000 mgr.smithi006.odexej (mgr.14176) 1425 : cluster [DBG] pgmap v978: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:24.519 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:24 smithi006 bash[26022]: audit 2023-11-17T06:04:22.896379+0000 mgr.smithi006.odexej (mgr.14176) 1426 : audit [DBG] from='client.15254 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:24 smithi156 bash[29655]: cluster 2023-11-17T06:04:22.624008+0000 mgr.smithi006.odexej (mgr.14176) 1425 : cluster [DBG] pgmap v978: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:24 smithi156 bash[29655]: audit 2023-11-17T06:04:22.896379+0000 mgr.smithi006.odexej (mgr.14176) 1426 : audit [DBG] from='client.15254 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:26.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:26 smithi006 bash[26022]: cluster 2023-11-17T06:04:24.625372+0000 mgr.smithi006.odexej (mgr.14176) 1427 : cluster [DBG] pgmap v979: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:26.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:26 smithi156 bash[29655]: cluster 2023-11-17T06:04:24.625372+0000 mgr.smithi006.odexej (mgr.14176) 1427 : cluster [DBG] pgmap v979: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:27.754 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:28.347 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:28.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:28 smithi006 bash[26022]: cluster 2023-11-17T06:04:26.626432+0000 mgr.smithi006.odexej (mgr.14176) 1428 : cluster [DBG] pgmap v980: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:28 smithi156 bash[29655]: cluster 2023-11-17T06:04:26.626432+0000 mgr.smithi006.odexej (mgr.14176) 1428 : cluster [DBG] pgmap v980: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:29.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:29.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:29 smithi006 bash[26022]: audit 2023-11-17T06:04:27.740728+0000 mgr.smithi006.odexej (mgr.14176) 1429 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:29.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:29 smithi156 bash[29655]: audit 2023-11-17T06:04:27.740728+0000 mgr.smithi006.odexej (mgr.14176) 1429 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:30.589 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:30 smithi006 bash[26022]: cluster 2023-11-17T06:04:28.627667+0000 mgr.smithi006.odexej (mgr.14176) 1430 : cluster [DBG] pgmap v981: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:30.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:30 smithi156 bash[29655]: cluster 2023-11-17T06:04:28.627667+0000 mgr.smithi006.odexej (mgr.14176) 1430 : cluster [DBG] pgmap v981: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:31.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:31 smithi006 bash[26022]: cluster 2023-11-17T06:04:30.629030+0000 mgr.smithi006.odexej (mgr.14176) 1431 : cluster [DBG] pgmap v982: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:31.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:31 smithi156 bash[29655]: cluster 2023-11-17T06:04:30.629030+0000 mgr.smithi006.odexej (mgr.14176) 1431 : cluster [DBG] pgmap v982: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:32.401 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:41.253 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:02:30.328208Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.umcpym on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:41.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:33 smithi156 bash[29655]: audit 2023-11-17T06:04:32.384451+0000 mgr.smithi006.odexej (mgr.14176) 1432 : audit [DBG] from='client.15262 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:41.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:33 smithi156 bash[29655]: cluster 2023-11-17T06:04:32.630161+0000 mgr.smithi006.odexej (mgr.14176) 1433 : cluster [DBG] pgmap v983: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:41.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:35 smithi156 bash[29655]: cluster 2023-11-17T06:04:34.631560+0000 mgr.smithi006.odexej (mgr.14176) 1434 : cluster [DBG] pgmap v984: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:37 smithi156 bash[29655]: cluster 2023-11-17T06:04:36.632666+0000 mgr.smithi006.odexej (mgr.14176) 1435 : cluster [DBG] pgmap v985: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.256 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:38 smithi156 bash[29655]: audit 2023-11-17T06:04:38.600077+0000 mon.smithi006 (mon.0) 1017 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: cluster 2023-11-17T06:04:38.633850+0000 mgr.smithi006.odexej (mgr.14176) 1436 : cluster [DBG] pgmap v986: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: audit 2023-11-17T06:04:38.907624+0000 mon.smithi006 (mon.0) 1018 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: audit 2023-11-17T06:04:38.909132+0000 mon.smithi006 (mon.0) 1019 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: audit 2023-11-17T06:04:38.916547+0000 mon.smithi006 (mon.0) 1020 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: cluster 2023-11-17T06:04:38.917439+0000 mgr.smithi006.odexej (mgr.14176) 1437 : cluster [DBG] pgmap v987: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: audit 2023-11-17T06:04:38.923795+0000 mon.smithi006 (mon.0) 1021 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: audit 2023-11-17T06:04:38.940357+0000 mon.smithi006 (mon.0) 1022 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:39 smithi156 bash[29655]: cephadm 2023-11-17T06:04:38.946551+0000 mgr.smithi006.odexej (mgr.14176) 1438 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qzntfs on smithi156 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:40 smithi156 bash[29655]: cluster 2023-11-17T06:04:39.914151+0000 mon.smithi006 (mon.0) 1023 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:04:41.257 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:40 smithi156 bash[29655]: cluster 2023-11-17T06:04:39.914222+0000 mon.smithi006 (mon.0) 1024 : cluster [INF] Cluster is now healthy 2023-11-17T06:04:41.258 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:41.261 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:33 smithi006 bash[26022]: audit 2023-11-17T06:04:32.384451+0000 mgr.smithi006.odexej (mgr.14176) 1432 : audit [DBG] from='client.15262 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:33 smithi006 bash[26022]: cluster 2023-11-17T06:04:32.630161+0000 mgr.smithi006.odexej (mgr.14176) 1433 : cluster [DBG] pgmap v983: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:35 smithi006 bash[26022]: cluster 2023-11-17T06:04:34.631560+0000 mgr.smithi006.odexej (mgr.14176) 1434 : cluster [DBG] pgmap v984: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:37 smithi006 bash[26022]: cluster 2023-11-17T06:04:36.632666+0000 mgr.smithi006.odexej (mgr.14176) 1435 : cluster [DBG] pgmap v985: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:38 smithi006 bash[26022]: audit 2023-11-17T06:04:38.600077+0000 mon.smithi006 (mon.0) 1017 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: cluster 2023-11-17T06:04:38.633850+0000 mgr.smithi006.odexej (mgr.14176) 1436 : cluster [DBG] pgmap v986: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:41.287 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: audit 2023-11-17T06:04:38.907624+0000 mon.smithi006 (mon.0) 1018 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:04:41.288 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: audit 2023-11-17T06:04:38.909132+0000 mon.smithi006 (mon.0) 1019 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:04:41.288 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: audit 2023-11-17T06:04:38.916547+0000 mon.smithi006 (mon.0) 1020 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:41.288 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: cluster 2023-11-17T06:04:38.917439+0000 mgr.smithi006.odexej (mgr.14176) 1437 : cluster [DBG] pgmap v987: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:41.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: audit 2023-11-17T06:04:38.923795+0000 mon.smithi006 (mon.0) 1021 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:04:41.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: audit 2023-11-17T06:04:38.940357+0000 mon.smithi006 (mon.0) 1022 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:41.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:39 smithi006 bash[26022]: cephadm 2023-11-17T06:04:38.946551+0000 mgr.smithi006.odexej (mgr.14176) 1438 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.qzntfs on smithi156 2023-11-17T06:04:41.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:40 smithi006 bash[26022]: cluster 2023-11-17T06:04:39.914151+0000 mon.smithi006 (mon.0) 1023 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:04:41.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:40 smithi006 bash[26022]: cluster 2023-11-17T06:04:39.914222+0000 mon.smithi006 (mon.0) 1024 : cluster [INF] Cluster is now healthy 2023-11-17T06:04:42.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:41 smithi006 bash[26022]: cluster 2023-11-17T06:04:40.918562+0000 mgr.smithi006.odexej (mgr.14176) 1439 : cluster [DBG] pgmap v988: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:42.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:41 smithi156 bash[29655]: cluster 2023-11-17T06:04:40.918562+0000 mgr.smithi006.odexej (mgr.14176) 1439 : cluster [DBG] pgmap v988: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:42.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:43.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:42 smithi006 bash[26022]: audit 2023-11-17T06:04:42.907050+0000 mon.smithi006 (mon.0) 1025 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qzntfs"}]: dispatch 2023-11-17T06:04:43.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:42 smithi156 bash[29655]: audit 2023-11-17T06:04:42.907050+0000 mon.smithi006 (mon.0) 1025 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.qzntfs"}]: dispatch 2023-11-17T06:04:44.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: cephadm 2023-11-17T06:04:42.905808+0000 mgr.smithi006.odexej (mgr.14176) 1440 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: cephadm 2023-11-17T06:04:42.906257+0000 mgr.smithi006.odexej (mgr.14176) 1441 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: cephadm 2023-11-17T06:04:42.908857+0000 mgr.smithi006.odexej (mgr.14176) 1442 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:45.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: cephadm 2023-11-17T06:04:42.913724+0000 mgr.smithi006.odexej (mgr.14176) 1443 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.rnshgv on smithi006 2023-11-17T06:04:45.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:43 smithi006 bash[26022]: cluster 2023-11-17T06:04:42.919591+0000 mgr.smithi006.odexej (mgr.14176) 1444 : cluster [DBG] pgmap v989: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:45.355 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: cephadm 2023-11-17T06:04:42.905808+0000 mgr.smithi006.odexej (mgr.14176) 1440 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.427 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.427 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.427 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.427 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.427 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:04:45.506 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.507 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: cephadm 2023-11-17T06:04:42.906257+0000 mgr.smithi006.odexej (mgr.14176) 1441 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: cephadm 2023-11-17T06:04:42.908857+0000 mgr.smithi006.odexej (mgr.14176) 1442 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.qzntfs ... 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: cephadm 2023-11-17T06:04:42.913724+0000 mgr.smithi006.odexej (mgr.14176) 1443 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.rnshgv on smithi006 2023-11-17T06:04:45.508 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:43 smithi156 bash[29655]: cluster 2023-11-17T06:04:42.919591+0000 mgr.smithi006.odexej (mgr.14176) 1444 : cluster [DBG] pgmap v989: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:45.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:45.509 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:34.741359Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dygrfo on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dygrfo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dygrfo\nDeploy daemon haproxy.nfs.foo.smithi156.dygrfo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.588857Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.mrqief on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-mrqief\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.mrqief\nDeploy daemon haproxy.nfs.foo.smithi006.mrqief ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:03:38.593311Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.tomgwz on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:45.950 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:46.307 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:45 smithi006 bash[26022]: cluster 2023-11-17T06:04:44.920805+0000 mgr.smithi006.odexej (mgr.14176) 1445 : cluster [DBG] pgmap v990: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:46.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:45 smithi156 bash[29655]: cluster 2023-11-17T06:04:44.920805+0000 mgr.smithi006.odexej (mgr.14176) 1445 : cluster [DBG] pgmap v990: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:46.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:47.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:46 smithi006 bash[26022]: audit 2023-11-17T06:04:45.359359+0000 mgr.smithi006.odexej (mgr.14176) 1446 : audit [DBG] from='client.15266 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:47.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:46 smithi006 bash[26022]: audit 2023-11-17T06:04:46.887064+0000 mon.smithi006 (mon.0) 1026 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.rnshgv"}]: dispatch 2023-11-17T06:04:47.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:46 smithi156 bash[29655]: audit 2023-11-17T06:04:45.359359+0000 mgr.smithi006.odexej (mgr.14176) 1446 : audit [DBG] from='client.15266 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:47.500 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:46 smithi156 bash[29655]: audit 2023-11-17T06:04:46.887064+0000 mon.smithi006 (mon.0) 1026 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.rnshgv"}]: dispatch 2023-11-17T06:04:48.247 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cephadm 2023-11-17T06:04:46.886197+0000 mgr.smithi006.odexej (mgr.14176) 1447 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cephadm 2023-11-17T06:04:46.886439+0000 mgr.smithi006.odexej (mgr.14176) 1448 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cephadm 2023-11-17T06:04:46.888868+0000 mgr.smithi006.odexej (mgr.14176) 1449 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:04:48.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cephadm 2023-11-17T06:04:46.893507+0000 mgr.smithi006.odexej (mgr.14176) 1450 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cephadm 2023-11-17T06:04:46.897370+0000 mgr.smithi006.odexej (mgr.14176) 1451 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cluster 2023-11-17T06:04:46.899475+0000 mgr.smithi006.odexej (mgr.14176) 1452 : cluster [DBG] pgmap v991: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:48.251 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:47 smithi006 bash[26022]: cluster 2023-11-17T06:04:47.768828+0000 mon.smithi006 (mon.0) 1027 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cephadm 2023-11-17T06:04:46.886197+0000 mgr.smithi006.odexej (mgr.14176) 1447 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cephadm 2023-11-17T06:04:46.886439+0000 mgr.smithi006.odexej (mgr.14176) 1448 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cephadm 2023-11-17T06:04:46.888868+0000 mgr.smithi006.odexej (mgr.14176) 1449 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.rnshgv ... 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:04:48.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cephadm 2023-11-17T06:04:46.893507+0000 mgr.smithi006.odexej (mgr.14176) 1450 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cephadm 2023-11-17T06:04:46.897370+0000 mgr.smithi006.odexej (mgr.14176) 1451 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cluster 2023-11-17T06:04:46.899475+0000 mgr.smithi006.odexej (mgr.14176) 1452 : cluster [DBG] pgmap v991: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:48.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:47 smithi156 bash[29655]: cluster 2023-11-17T06:04:47.768828+0000 mon.smithi006 (mon.0) 1027 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:04:50.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:49 smithi006 bash[26022]: cluster 2023-11-17T06:04:48.900747+0000 mgr.smithi006.odexej (mgr.14176) 1453 : cluster [DBG] pgmap v992: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:50.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:50.246 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.888529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\nDeploy daemon haproxy.nfs.foo.smithi006.rnshgv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.893294Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:50.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:49 smithi156 bash[29655]: cluster 2023-11-17T06:04:48.900747+0000 mgr.smithi006.odexej (mgr.14176) 1453 : cluster [DBG] pgmap v992: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:04:50.899 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:51.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:51 smithi006 bash[26022]: audit 2023-11-17T06:04:50.231095+0000 mgr.smithi006.odexej (mgr.14176) 1454 : audit [DBG] from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:51.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:51 smithi156 bash[29655]: audit 2023-11-17T06:04:50.231095+0000 mgr.smithi006.odexej (mgr.14176) 1454 : audit [DBG] from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:51.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:52.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:52 smithi006 bash[26022]: cluster 2023-11-17T06:04:50.902529+0000 mgr.smithi006.odexej (mgr.14176) 1455 : cluster [DBG] pgmap v993: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:52.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:52 smithi006 bash[26022]: audit 2023-11-17T06:04:51.153755+0000 mon.smithi006 (mon.0) 1028 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:52.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:52 smithi156 bash[29655]: cluster 2023-11-17T06:04:50.902529+0000 mgr.smithi006.odexej (mgr.14176) 1455 : cluster [DBG] pgmap v993: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:52.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:52 smithi156 bash[29655]: audit 2023-11-17T06:04:51.153755+0000 mon.smithi006 (mon.0) 1028 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:04:54.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:54 smithi006 bash[26022]: cluster 2023-11-17T06:04:52.903624+0000 mgr.smithi006.odexej (mgr.14176) 1456 : cluster [DBG] pgmap v994: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:54.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:54 smithi156 bash[29655]: cluster 2023-11-17T06:04:52.903624+0000 mgr.smithi006.odexej (mgr.14176) 1456 : cluster [DBG] pgmap v994: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:04:55.233 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:55.234 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.888529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\nDeploy daemon haproxy.nfs.foo.smithi006.rnshgv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.893294Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:04:55.779 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:04:56.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:56 smithi156 bash[29655]: cluster 2023-11-17T06:04:54.905337+0000 mgr.smithi006.odexej (mgr.14176) 1457 : cluster [DBG] pgmap v995: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:56 smithi156 bash[29655]: audit 2023-11-17T06:04:55.220448+0000 mgr.smithi006.odexej (mgr.14176) 1458 : audit [DBG] from='client.15274 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:56.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:56 smithi006 bash[26022]: cluster 2023-11-17T06:04:54.905337+0000 mgr.smithi006.odexej (mgr.14176) 1457 : cluster [DBG] pgmap v995: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:56.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:56 smithi006 bash[26022]: audit 2023-11-17T06:04:55.220448+0000 mgr.smithi006.odexej (mgr.14176) 1458 : audit [DBG] from='client.15274 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:04:56.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:04:58.074 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:57 smithi006 bash[26022]: cluster 2023-11-17T06:04:56.906411+0000 mgr.smithi006.odexej (mgr.14176) 1459 : cluster [DBG] pgmap v996: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:58.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:57 smithi156 bash[29655]: cluster 2023-11-17T06:04:56.906411+0000 mgr.smithi006.odexej (mgr.14176) 1459 : cluster [DBG] pgmap v996: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:04:59.830 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:04:59.830 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.888529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\nDeploy daemon haproxy.nfs.foo.smithi006.rnshgv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.893294Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:05:00.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:04:59 smithi006 bash[26022]: cluster 2023-11-17T06:04:58.907689+0000 mgr.smithi006.odexej (mgr.14176) 1460 : cluster [DBG] pgmap v997: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:00.385 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:05:00.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:04:59 smithi156 bash[29655]: cluster 2023-11-17T06:04:58.907689+0000 mgr.smithi006.odexej (mgr.14176) 1460 : cluster [DBG] pgmap v997: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:01.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:05:01.398 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:00 smithi006 bash[26022]: audit 2023-11-17T06:04:59.819140+0000 mgr.smithi006.odexej (mgr.14176) 1461 : audit [DBG] from='client.15278 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:05:01.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:00 smithi156 bash[29655]: audit 2023-11-17T06:04:59.819140+0000 mgr.smithi006.odexej (mgr.14176) 1461 : audit [DBG] from='client.15278 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:05:02.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:01 smithi006 bash[26022]: cluster 2023-11-17T06:05:00.909132+0000 mgr.smithi006.odexej (mgr.14176) 1462 : cluster [DBG] pgmap v998: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:02.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:01 smithi156 bash[29655]: cluster 2023-11-17T06:05:00.909132+0000 mgr.smithi006.odexej (mgr.14176) 1462 : cluster [DBG] pgmap v998: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:04.304 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:03 smithi006 bash[26022]: cluster 2023-11-17T06:05:02.910220+0000 mgr.smithi006.odexej (mgr.14176) 1463 : cluster [DBG] pgmap v999: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:04.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:03 smithi156 bash[29655]: cluster 2023-11-17T06:05:02.910220+0000 mgr.smithi006.odexej (mgr.14176) 1463 : cluster [DBG] pgmap v999: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:05:04.668 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.888529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\nDeploy daemon haproxy.nfs.foo.smithi006.rnshgv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.893294Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:05:05.291 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:05:06.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:05:30.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:05 smithi156 bash[29655]: audit 2023-11-17T06:05:04.656791+0000 mgr.smithi006.odexej (mgr.14176) 1464 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:05:30.095 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:05 smithi156 bash[29655]: cluster 2023-11-17T06:05:04.911704+0000 mgr.smithi006.odexej (mgr.14176) 1465 : cluster [DBG] pgmap v1000: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.214 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:07 smithi156 bash[29655]: cluster 2023-11-17T06:05:06.912752+0000 mgr.smithi006.odexej (mgr.14176) 1466 : cluster [DBG] pgmap v1001: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.214 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:09 smithi156 bash[29655]: cluster 2023-11-17T06:05:08.913919+0000 mgr.smithi006.odexej (mgr.14176) 1467 : cluster [DBG] pgmap v1002: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.214 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:11 smithi156 bash[29655]: cluster 2023-11-17T06:05:10.915385+0000 mgr.smithi006.odexej (mgr.14176) 1468 : cluster [DBG] pgmap v1003: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.214 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:13 smithi156 bash[29655]: cluster 2023-11-17T06:05:12.916482+0000 mgr.smithi006.odexej (mgr.14176) 1469 : cluster [DBG] pgmap v1004: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:15 smithi156 bash[29655]: cluster 2023-11-17T06:05:14.917847+0000 mgr.smithi006.odexej (mgr.14176) 1470 : cluster [DBG] pgmap v1005: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:17 smithi156 bash[29655]: cluster 2023-11-17T06:05:16.918911+0000 mgr.smithi006.odexej (mgr.14176) 1471 : cluster [DBG] pgmap v1006: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:19 smithi156 bash[29655]: cluster 2023-11-17T06:05:18.920091+0000 mgr.smithi006.odexej (mgr.14176) 1472 : cluster [DBG] pgmap v1007: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:21 smithi156 bash[29655]: cluster 2023-11-17T06:05:20.921438+0000 mgr.smithi006.odexej (mgr.14176) 1473 : cluster [DBG] pgmap v1008: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:23 smithi156 bash[29655]: cluster 2023-11-17T06:05:22.922481+0000 mgr.smithi006.odexej (mgr.14176) 1474 : cluster [DBG] pgmap v1009: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:25 smithi156 bash[29655]: cluster 2023-11-17T06:05:24.923942+0000 mgr.smithi006.odexej (mgr.14176) 1475 : cluster [DBG] pgmap v1010: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:27 smithi156 bash[29655]: cluster 2023-11-17T06:05:26.924990+0000 mgr.smithi006.odexej (mgr.14176) 1476 : cluster [DBG] pgmap v1011: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:05 smithi006 bash[26022]: audit 2023-11-17T06:05:04.656791+0000 mgr.smithi006.odexej (mgr.14176) 1464 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:05 smithi006 bash[26022]: cluster 2023-11-17T06:05:04.911704+0000 mgr.smithi006.odexej (mgr.14176) 1465 : cluster [DBG] pgmap v1000: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:07 smithi006 bash[26022]: cluster 2023-11-17T06:05:06.912752+0000 mgr.smithi006.odexej (mgr.14176) 1466 : cluster [DBG] pgmap v1001: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:09 smithi006 bash[26022]: cluster 2023-11-17T06:05:08.913919+0000 mgr.smithi006.odexej (mgr.14176) 1467 : cluster [DBG] pgmap v1002: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:11 smithi006 bash[26022]: cluster 2023-11-17T06:05:10.915385+0000 mgr.smithi006.odexej (mgr.14176) 1468 : cluster [DBG] pgmap v1003: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:13 smithi006 bash[26022]: cluster 2023-11-17T06:05:12.916482+0000 mgr.smithi006.odexej (mgr.14176) 1469 : cluster [DBG] pgmap v1004: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:15 smithi006 bash[26022]: cluster 2023-11-17T06:05:14.917847+0000 mgr.smithi006.odexej (mgr.14176) 1470 : cluster [DBG] pgmap v1005: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:17 smithi006 bash[26022]: cluster 2023-11-17T06:05:16.918911+0000 mgr.smithi006.odexej (mgr.14176) 1471 : cluster [DBG] pgmap v1006: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:19 smithi006 bash[26022]: cluster 2023-11-17T06:05:18.920091+0000 mgr.smithi006.odexej (mgr.14176) 1472 : cluster [DBG] pgmap v1007: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:21 smithi006 bash[26022]: cluster 2023-11-17T06:05:20.921438+0000 mgr.smithi006.odexej (mgr.14176) 1473 : cluster [DBG] pgmap v1008: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:23 smithi006 bash[26022]: cluster 2023-11-17T06:05:22.922481+0000 mgr.smithi006.odexej (mgr.14176) 1474 : cluster [DBG] pgmap v1009: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:25 smithi006 bash[26022]: cluster 2023-11-17T06:05:24.923942+0000 mgr.smithi006.odexej (mgr.14176) 1475 : cluster [DBG] pgmap v1010: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:27 smithi006 bash[26022]: cluster 2023-11-17T06:05:26.924990+0000 mgr.smithi006.odexej (mgr.14176) 1476 : cluster [DBG] pgmap v1011: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:29 smithi006 bash[26022]: cluster 2023-11-17T06:05:28.926280+0000 mgr.smithi006.odexej (mgr.14176) 1477 : cluster [DBG] pgmap v1012: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:30.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:29 smithi156 bash[29655]: cluster 2023-11-17T06:05:28.926280+0000 mgr.smithi006.odexej (mgr.14176) 1477 : cluster [DBG] pgmap v1012: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:32.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:31 smithi006 bash[26022]: cluster 2023-11-17T06:05:30.927769+0000 mgr.smithi006.odexej (mgr.14176) 1478 : cluster [DBG] pgmap v1013: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:32.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:31 smithi156 bash[29655]: cluster 2023-11-17T06:05:30.927769+0000 mgr.smithi006.odexej (mgr.14176) 1478 : cluster [DBG] pgmap v1013: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:05:33.292 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:28.774 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:03:38.595450Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.zengpb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:42.908572Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.qzntfs on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-qzntfs\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.qzntfs\nDeploy daemon haproxy.nfs.foo.smithi156.qzntfs ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.888529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.rnshgv on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-rnshgv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.rnshgv\nDeploy daemon haproxy.nfs.foo.smithi006.rnshgv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:04:46.893294Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nlcfcc on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:28.777 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:28.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:33 smithi156 bash[29655]: cluster 2023-11-17T06:05:32.929160+0000 mgr.smithi006.odexej (mgr.14176) 1479 : cluster [DBG] pgmap v1014: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:28.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:35 smithi156 bash[29655]: audit 2023-11-17T06:05:33.282393+0000 mgr.smithi006.odexej (mgr.14176) 1480 : audit [DBG] from='client.15286 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:28.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:36 smithi156 bash[29655]: cluster 2023-11-17T06:05:34.930603+0000 mgr.smithi006.odexej (mgr.14176) 1481 : cluster [DBG] pgmap v1015: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:37 smithi156 bash[29655]: cluster 2023-11-17T06:05:36.931672+0000 mgr.smithi006.odexej (mgr.14176) 1482 : cluster [DBG] pgmap v1016: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:39 smithi156 bash[29655]: cluster 2023-11-17T06:05:38.932809+0000 mgr.smithi006.odexej (mgr.14176) 1483 : cluster [DBG] pgmap v1017: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.788 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:41 smithi156 bash[29655]: cluster 2023-11-17T06:05:40.934273+0000 mgr.smithi006.odexej (mgr.14176) 1484 : cluster [DBG] pgmap v1018: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.788 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:44 smithi156 bash[29655]: cluster 2023-11-17T06:05:42.935371+0000 mgr.smithi006.odexej (mgr.14176) 1485 : cluster [DBG] pgmap v1019: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:28.788 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:46 smithi156 bash[29655]: cluster 2023-11-17T06:05:44.936799+0000 mgr.smithi006.odexej (mgr.14176) 1486 : cluster [DBG] pgmap v1020: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.821 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:46.905086+0000 mon.smithi006 (mon.0) 1029 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: cluster 2023-11-17T06:05:46.937704+0000 mgr.smithi006.odexej (mgr.14176) 1487 : cluster [DBG] pgmap v1021: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.215011+0000 mon.smithi006 (mon.0) 1030 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.509674+0000 mon.smithi006 (mon.0) 1031 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.511748+0000 mon.smithi006 (mon.0) 1032 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.522453+0000 mon.smithi006 (mon.0) 1033 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.533957+0000 mon.smithi006 (mon.0) 1034 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:47 smithi156 bash[29655]: audit 2023-11-17T06:05:47.553577+0000 mon.smithi006 (mon.0) 1035 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:48 smithi156 bash[29655]: cluster 2023-11-17T06:05:47.523816+0000 mgr.smithi006.odexej (mgr.14176) 1488 : cluster [DBG] pgmap v1022: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:48 smithi156 bash[29655]: cephadm 2023-11-17T06:05:47.557289+0000 mgr.smithi006.odexej (mgr.14176) 1489 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.kgzbrn on smithi156 2023-11-17T06:06:28.964 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:48 smithi156 bash[29655]: cluster 2023-11-17T06:05:47.787833+0000 mon.smithi006 (mon.0) 1036 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:48 smithi156 bash[29655]: cluster 2023-11-17T06:05:47.787898+0000 mon.smithi006 (mon.0) 1037 : cluster [INF] Cluster is now healthy 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:50 smithi156 bash[29655]: cluster 2023-11-17T06:05:49.524957+0000 mgr.smithi006.odexej (mgr.14176) 1490 : cluster [DBG] pgmap v1023: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:50 smithi156 bash[29655]: audit 2023-11-17T06:05:49.742028+0000 mon.smithi006 (mon.0) 1038 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:51 smithi156 bash[29655]: audit 2023-11-17T06:05:51.316854+0000 mon.smithi006 (mon.0) 1039 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.kgzbrn"}]: dispatch 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: cephadm 2023-11-17T06:05:51.315401+0000 mgr.smithi006.odexej (mgr.14176) 1491 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:28.965 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.966 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: cephadm 2023-11-17T06:05:51.316070+0000 mgr.smithi006.odexej (mgr.14176) 1492 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: cephadm 2023-11-17T06:05:51.318610+0000 mgr.smithi006.odexej (mgr.14176) 1493 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:28.967 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:28.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:28.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:28.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:28.968 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:28.998 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: cephadm 2023-11-17T06:05:51.324641+0000 mgr.smithi006.odexej (mgr.14176) 1494 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.hhjtxc on smithi006 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:52 smithi156 bash[29655]: cluster 2023-11-17T06:05:51.525560+0000 mgr.smithi006.odexej (mgr.14176) 1495 : cluster [DBG] pgmap v1024: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:54 smithi156 bash[29655]: cluster 2023-11-17T06:05:53.526460+0000 mgr.smithi006.odexej (mgr.14176) 1496 : cluster [DBG] pgmap v1025: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cephadm 2023-11-17T06:05:55.061312+0000 mgr.smithi006.odexej (mgr.14176) 1497 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.046 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.047 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.048 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.106 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cephadm 2023-11-17T06:05:55.061540+0000 mgr.smithi006.odexej (mgr.14176) 1498 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: audit 2023-11-17T06:05:55.062065+0000 mon.smithi006 (mon.0) 1040 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.hhjtxc"}]: dispatch 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cephadm 2023-11-17T06:05:55.063273+0000 mgr.smithi006.odexej (mgr.14176) 1499 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.107 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cephadm 2023-11-17T06:05:55.065499+0000 mgr.smithi006.odexej (mgr.14176) 1500 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cephadm 2023-11-17T06:05:55.069220+0000 mgr.smithi006.odexej (mgr.14176) 1501 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:55 smithi156 bash[29655]: cluster 2023-11-17T06:05:55.070883+0000 mgr.smithi006.odexej (mgr.14176) 1502 : cluster [DBG] pgmap v1026: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:56 smithi156 bash[29655]: cluster 2023-11-17T06:05:55.799352+0000 mon.smithi006 (mon.0) 1041 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:56 smithi156 bash[29655]: audit 2023-11-17T06:05:56.206587+0000 mon.smithi006 (mon.0) 1042 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:05:57 smithi156 bash[29655]: cluster 2023-11-17T06:05:57.072077+0000 mgr.smithi006.odexej (mgr.14176) 1503 : cluster [DBG] pgmap v1027: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:00 smithi156 bash[29655]: cluster 2023-11-17T06:05:59.073258+0000 mgr.smithi006.odexej (mgr.14176) 1504 : cluster [DBG] pgmap v1028: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:02 smithi156 bash[29655]: cluster 2023-11-17T06:06:01.074583+0000 mgr.smithi006.odexej (mgr.14176) 1505 : cluster [DBG] pgmap v1029: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:04 smithi156 bash[29655]: cluster 2023-11-17T06:06:03.075742+0000 mgr.smithi006.odexej (mgr.14176) 1506 : cluster [DBG] pgmap v1030: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:05 smithi156 bash[29655]: audit 2023-11-17T06:06:04.741733+0000 mon.smithi006 (mon.0) 1043 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:05 smithi156 bash[29655]: cluster 2023-11-17T06:06:05.077043+0000 mgr.smithi006.odexej (mgr.14176) 1507 : cluster [DBG] pgmap v1031: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:07 smithi156 bash[29655]: cluster 2023-11-17T06:06:07.078037+0000 mgr.smithi006.odexej (mgr.14176) 1508 : cluster [DBG] pgmap v1032: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:10 smithi156 bash[29655]: cluster 2023-11-17T06:06:09.079169+0000 mgr.smithi006.odexej (mgr.14176) 1509 : cluster [DBG] pgmap v1033: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.108 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:12 smithi156 bash[29655]: cluster 2023-11-17T06:06:11.080504+0000 mgr.smithi006.odexej (mgr.14176) 1510 : cluster [DBG] pgmap v1034: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.110 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:14 smithi156 bash[29655]: cluster 2023-11-17T06:06:13.081554+0000 mgr.smithi006.odexej (mgr.14176) 1511 : cluster [DBG] pgmap v1035: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:29.221 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:16 smithi156 bash[29655]: cluster 2023-11-17T06:06:15.082966+0000 mgr.smithi006.odexej (mgr.14176) 1512 : cluster [DBG] pgmap v1036: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.221 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:17 smithi156 bash[29655]: cluster 2023-11-17T06:06:17.084006+0000 mgr.smithi006.odexej (mgr.14176) 1513 : cluster [DBG] pgmap v1037: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.221 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:20 smithi156 bash[29655]: cluster 2023-11-17T06:06:19.085161+0000 mgr.smithi006.odexej (mgr.14176) 1514 : cluster [DBG] pgmap v1038: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.221 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:22 smithi156 bash[29655]: cluster 2023-11-17T06:06:21.086458+0000 mgr.smithi006.odexej (mgr.14176) 1515 : cluster [DBG] pgmap v1039: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.221 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:24 smithi156 bash[29655]: cluster 2023-11-17T06:06:23.087475+0000 mgr.smithi006.odexej (mgr.14176) 1516 : cluster [DBG] pgmap v1040: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:29.222 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:26 smithi156 bash[29655]: cluster 2023-11-17T06:06:25.088836+0000 mgr.smithi006.odexej (mgr.14176) 1517 : cluster [DBG] pgmap v1041: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.222 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:27 smithi156 bash[29655]: cluster 2023-11-17T06:06:27.089846+0000 mgr.smithi006.odexej (mgr.14176) 1518 : cluster [DBG] pgmap v1042: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:33 smithi006 bash[26022]: cluster 2023-11-17T06:05:32.929160+0000 mgr.smithi006.odexej (mgr.14176) 1479 : cluster [DBG] pgmap v1014: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:29.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:35 smithi006 bash[26022]: audit 2023-11-17T06:05:33.282393+0000 mgr.smithi006.odexej (mgr.14176) 1480 : audit [DBG] from='client.15286 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:29.223 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:36 smithi006 bash[26022]: cluster 2023-11-17T06:05:34.930603+0000 mgr.smithi006.odexej (mgr.14176) 1481 : cluster [DBG] pgmap v1015: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:37 smithi006 bash[26022]: cluster 2023-11-17T06:05:36.931672+0000 mgr.smithi006.odexej (mgr.14176) 1482 : cluster [DBG] pgmap v1016: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:39 smithi006 bash[26022]: cluster 2023-11-17T06:05:38.932809+0000 mgr.smithi006.odexej (mgr.14176) 1483 : cluster [DBG] pgmap v1017: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:41 smithi006 bash[26022]: cluster 2023-11-17T06:05:40.934273+0000 mgr.smithi006.odexej (mgr.14176) 1484 : cluster [DBG] pgmap v1018: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:44 smithi006 bash[26022]: cluster 2023-11-17T06:05:42.935371+0000 mgr.smithi006.odexej (mgr.14176) 1485 : cluster [DBG] pgmap v1019: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:46 smithi006 bash[26022]: cluster 2023-11-17T06:05:44.936799+0000 mgr.smithi006.odexej (mgr.14176) 1486 : cluster [DBG] pgmap v1020: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:46.905086+0000 mon.smithi006 (mon.0) 1029 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: cluster 2023-11-17T06:05:46.937704+0000 mgr.smithi006.odexej (mgr.14176) 1487 : cluster [DBG] pgmap v1021: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.215011+0000 mon.smithi006 (mon.0) 1030 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.509674+0000 mon.smithi006 (mon.0) 1031 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.511748+0000 mon.smithi006 (mon.0) 1032 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.522453+0000 mon.smithi006 (mon.0) 1033 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.533957+0000 mon.smithi006 (mon.0) 1034 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:06:29.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:47 smithi006 bash[26022]: audit 2023-11-17T06:05:47.553577+0000 mon.smithi006 (mon.0) 1035 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:48 smithi006 bash[26022]: cluster 2023-11-17T06:05:47.523816+0000 mgr.smithi006.odexej (mgr.14176) 1488 : cluster [DBG] pgmap v1022: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.228 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:48 smithi006 bash[26022]: cephadm 2023-11-17T06:05:47.557289+0000 mgr.smithi006.odexej (mgr.14176) 1489 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.kgzbrn on smithi156 2023-11-17T06:06:29.228 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:48 smithi006 bash[26022]: cluster 2023-11-17T06:05:47.787833+0000 mon.smithi006 (mon.0) 1036 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:06:29.228 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:48 smithi006 bash[26022]: cluster 2023-11-17T06:05:47.787898+0000 mon.smithi006 (mon.0) 1037 : cluster [INF] Cluster is now healthy 2023-11-17T06:06:29.228 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:50 smithi006 bash[26022]: cluster 2023-11-17T06:05:49.524957+0000 mgr.smithi006.odexej (mgr.14176) 1490 : cluster [DBG] pgmap v1023: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.313 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:50 smithi006 bash[26022]: audit 2023-11-17T06:05:49.742028+0000 mon.smithi006 (mon.0) 1038 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:29.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:51 smithi006 bash[26022]: audit 2023-11-17T06:05:51.316854+0000 mon.smithi006 (mon.0) 1039 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.kgzbrn"}]: dispatch 2023-11-17T06:06:29.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: cephadm 2023-11-17T06:05:51.315401+0000 mgr.smithi006.odexej (mgr.14176) 1491 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.656 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.882 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.882 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.882 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:29.882 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:06:29.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:06:29.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:06:29.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:06:29.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:06:29.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:29.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.886 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.886 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.889 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.933 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: cephadm 2023-11-17T06:05:51.316070+0000 mgr.smithi006.odexej (mgr.14176) 1492 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: cephadm 2023-11-17T06:05:51.318610+0000 mgr.smithi006.odexej (mgr.14176) 1493 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.934 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn 2023-11-17T06:06:29.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:29.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn 2023-11-17T06:06:29.935 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.kgzbrn ... 2023-11-17T06:06:29.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:29.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:29.936 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:29.937 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:29.937 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: cephadm 2023-11-17T06:05:51.324641+0000 mgr.smithi006.odexej (mgr.14176) 1494 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.hhjtxc on smithi006 2023-11-17T06:06:29.937 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:52 smithi006 bash[26022]: cluster 2023-11-17T06:05:51.525560+0000 mgr.smithi006.odexej (mgr.14176) 1495 : cluster [DBG] pgmap v1024: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.937 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:54 smithi006 bash[26022]: cluster 2023-11-17T06:05:53.526460+0000 mgr.smithi006.odexej (mgr.14176) 1496 : cluster [DBG] pgmap v1025: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:06:29.938 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cephadm 2023-11-17T06:05:55.061312+0000 mgr.smithi006.odexej (mgr.14176) 1497 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:29.939 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.221 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:30.221 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.221 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:30.281 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.282 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:30.282 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:30.282 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:30.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.315 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:06:30.316 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:06:30.317 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:06:30.344 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:06:30.344 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:06:30.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:30.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:30.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.346 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:30.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:30.347 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:30.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:30.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cephadm 2023-11-17T06:05:55.061540+0000 mgr.smithi006.odexej (mgr.14176) 1498 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: audit 2023-11-17T06:05:55.062065+0000 mon.smithi006 (mon.0) 1040 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.hhjtxc"}]: dispatch 2023-11-17T06:06:30.349 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cephadm 2023-11-17T06:05:55.063273+0000 mgr.smithi006.odexej (mgr.14176) 1499 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:30.349 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.349 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc 2023-11-17T06:06:30.349 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.349 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:06:30.350 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc 2023-11-17T06:06:30.350 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hhjtxc ... 2023-11-17T06:06:30.350 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.350 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:06:30.351 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:06:30.351 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:06:30.352 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:06:30.352 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cephadm 2023-11-17T06:05:55.065499+0000 mgr.smithi006.odexej (mgr.14176) 1500 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:06:30.352 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cephadm 2023-11-17T06:05:55.069220+0000 mgr.smithi006.odexej (mgr.14176) 1501 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:06:30.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:55 smithi006 bash[26022]: cluster 2023-11-17T06:05:55.070883+0000 mgr.smithi006.odexej (mgr.14176) 1502 : cluster [DBG] pgmap v1026: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:06:30.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:56 smithi006 bash[26022]: cluster 2023-11-17T06:05:55.799352+0000 mon.smithi006 (mon.0) 1041 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:06:30.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:56 smithi006 bash[26022]: audit 2023-11-17T06:05:56.206587+0000 mon.smithi006 (mon.0) 1042 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:30.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:05:57 smithi006 bash[26022]: cluster 2023-11-17T06:05:57.072077+0000 mgr.smithi006.odexej (mgr.14176) 1503 : cluster [DBG] pgmap v1027: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:06:30.354 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:00 smithi006 bash[26022]: cluster 2023-11-17T06:05:59.073258+0000 mgr.smithi006.odexej (mgr.14176) 1504 : cluster [DBG] pgmap v1028: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:30.354 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:02 smithi006 bash[26022]: cluster 2023-11-17T06:06:01.074583+0000 mgr.smithi006.odexej (mgr.14176) 1505 : cluster [DBG] pgmap v1029: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:30.354 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:04 smithi006 bash[26022]: cluster 2023-11-17T06:06:03.075742+0000 mgr.smithi006.odexej (mgr.14176) 1506 : cluster [DBG] pgmap v1030: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:06:30.354 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:05 smithi006 bash[26022]: audit 2023-11-17T06:06:04.741733+0000 mon.smithi006 (mon.0) 1043 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:30.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:05 smithi006 bash[26022]: cluster 2023-11-17T06:06:05.077043+0000 mgr.smithi006.odexej (mgr.14176) 1507 : cluster [DBG] pgmap v1031: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:06:30.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:07 smithi006 bash[26022]: cluster 2023-11-17T06:06:07.078037+0000 mgr.smithi006.odexej (mgr.14176) 1508 : cluster [DBG] pgmap v1032: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:10 smithi006 bash[26022]: cluster 2023-11-17T06:06:09.079169+0000 mgr.smithi006.odexej (mgr.14176) 1509 : cluster [DBG] pgmap v1033: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:12 smithi006 bash[26022]: cluster 2023-11-17T06:06:11.080504+0000 mgr.smithi006.odexej (mgr.14176) 1510 : cluster [DBG] pgmap v1034: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.355 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:14 smithi006 bash[26022]: cluster 2023-11-17T06:06:13.081554+0000 mgr.smithi006.odexej (mgr.14176) 1511 : cluster [DBG] pgmap v1035: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:30.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:16 smithi006 bash[26022]: cluster 2023-11-17T06:06:15.082966+0000 mgr.smithi006.odexej (mgr.14176) 1512 : cluster [DBG] pgmap v1036: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:17 smithi006 bash[26022]: cluster 2023-11-17T06:06:17.084006+0000 mgr.smithi006.odexej (mgr.14176) 1513 : cluster [DBG] pgmap v1037: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:20 smithi006 bash[26022]: cluster 2023-11-17T06:06:19.085161+0000 mgr.smithi006.odexej (mgr.14176) 1514 : cluster [DBG] pgmap v1038: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:22 smithi006 bash[26022]: cluster 2023-11-17T06:06:21.086458+0000 mgr.smithi006.odexej (mgr.14176) 1515 : cluster [DBG] pgmap v1039: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.357 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:24 smithi006 bash[26022]: cluster 2023-11-17T06:06:23.087475+0000 mgr.smithi006.odexej (mgr.14176) 1516 : cluster [DBG] pgmap v1040: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:30.357 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:26 smithi006 bash[26022]: cluster 2023-11-17T06:06:25.088836+0000 mgr.smithi006.odexej (mgr.14176) 1517 : cluster [DBG] pgmap v1041: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.357 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:27 smithi006 bash[26022]: cluster 2023-11-17T06:06:27.089846+0000 mgr.smithi006.odexej (mgr.14176) 1518 : cluster [DBG] pgmap v1042: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:30.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:30 smithi006 bash[26022]: cluster 2023-11-17T06:06:29.091008+0000 mgr.smithi006.odexej (mgr.14176) 1519 : cluster [DBG] pgmap v1043: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:30.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:30 smithi156 bash[29655]: cluster 2023-11-17T06:06:29.091008+0000 mgr.smithi006.odexej (mgr.14176) 1519 : cluster [DBG] pgmap v1043: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:32.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:32 smithi006 bash[26022]: cluster 2023-11-17T06:06:31.092029+0000 mgr.smithi006.odexej (mgr.14176) 1520 : cluster [DBG] pgmap v1044: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:32.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:32 smithi156 bash[29655]: cluster 2023-11-17T06:06:31.092029+0000 mgr.smithi006.odexej (mgr.14176) 1520 : cluster [DBG] pgmap v1044: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:33.456 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:33.456 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:34.070 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:34.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:34 smithi006 bash[26022]: cluster 2023-11-17T06:06:33.092773+0000 mgr.smithi006.odexej (mgr.14176) 1521 : cluster [DBG] pgmap v1045: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:34.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:34 smithi156 bash[29655]: cluster 2023-11-17T06:06:33.092773+0000 mgr.smithi006.odexej (mgr.14176) 1521 : cluster [DBG] pgmap v1045: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:35.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:35.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:35 smithi006 bash[26022]: audit 2023-11-17T06:06:33.446508+0000 mgr.smithi006.odexej (mgr.14176) 1522 : audit [DBG] from='client.15290 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:35.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:35 smithi156 bash[29655]: audit 2023-11-17T06:06:33.446508+0000 mgr.smithi006.odexej (mgr.14176) 1522 : audit [DBG] from='client.15290 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:36.375 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:36 smithi006 bash[26022]: cluster 2023-11-17T06:06:35.094144+0000 mgr.smithi006.odexej (mgr.14176) 1523 : cluster [DBG] pgmap v1046: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:36.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:36 smithi156 bash[29655]: cluster 2023-11-17T06:06:35.094144+0000 mgr.smithi006.odexej (mgr.14176) 1523 : cluster [DBG] pgmap v1046: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:37.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:37 smithi006 bash[26022]: cluster 2023-11-17T06:06:37.095207+0000 mgr.smithi006.odexej (mgr.14176) 1524 : cluster [DBG] pgmap v1047: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:38.069 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:38.069 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:38.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:37 smithi156 bash[29655]: cluster 2023-11-17T06:06:37.095207+0000 mgr.smithi006.odexej (mgr.14176) 1524 : cluster [DBG] pgmap v1047: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:38.672 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:39.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:38 smithi006 bash[26022]: audit 2023-11-17T06:06:38.054794+0000 mgr.smithi006.odexej (mgr.14176) 1525 : audit [DBG] from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:39.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:38 smithi156 bash[29655]: audit 2023-11-17T06:06:38.054794+0000 mgr.smithi006.odexej (mgr.14176) 1525 : audit [DBG] from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:39.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:40.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:39 smithi006 bash[26022]: cluster 2023-11-17T06:06:39.096444+0000 mgr.smithi006.odexej (mgr.14176) 1526 : cluster [DBG] pgmap v1048: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:40.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:39 smithi156 bash[29655]: cluster 2023-11-17T06:06:39.096444+0000 mgr.smithi006.odexej (mgr.14176) 1526 : cluster [DBG] pgmap v1048: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:42.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:42 smithi006 bash[26022]: cluster 2023-11-17T06:06:41.097293+0000 mgr.smithi006.odexej (mgr.14176) 1527 : cluster [DBG] pgmap v1049: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:42.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:42 smithi156 bash[29655]: cluster 2023-11-17T06:06:41.097293+0000 mgr.smithi006.odexej (mgr.14176) 1527 : cluster [DBG] pgmap v1049: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:42.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:42.637 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:43.218 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:44.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:44 smithi006 bash[26022]: audit 2023-11-17T06:06:42.622781+0000 mgr.smithi006.odexej (mgr.14176) 1528 : audit [DBG] from='client.15298 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:44 smithi006 bash[26022]: cluster 2023-11-17T06:06:43.098352+0000 mgr.smithi006.odexej (mgr.14176) 1529 : cluster [DBG] pgmap v1050: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:44.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:44 smithi156 bash[29655]: audit 2023-11-17T06:06:42.622781+0000 mgr.smithi006.odexej (mgr.14176) 1528 : audit [DBG] from='client.15298 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:44.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:44 smithi156 bash[29655]: cluster 2023-11-17T06:06:43.098352+0000 mgr.smithi006.odexej (mgr.14176) 1529 : cluster [DBG] pgmap v1050: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:46.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:46 smithi156 bash[29655]: cluster 2023-11-17T06:06:45.099446+0000 mgr.smithi006.odexej (mgr.14176) 1530 : cluster [DBG] pgmap v1051: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:46.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:46 smithi006 bash[26022]: cluster 2023-11-17T06:06:45.099446+0000 mgr.smithi006.odexej (mgr.14176) 1530 : cluster [DBG] pgmap v1051: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:47.287 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:47.288 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:47.883 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:48.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:47 smithi006 bash[26022]: cluster 2023-11-17T06:06:47.100575+0000 mgr.smithi006.odexej (mgr.14176) 1531 : cluster [DBG] pgmap v1052: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:48.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:47 smithi156 bash[29655]: cluster 2023-11-17T06:06:47.100575+0000 mgr.smithi006.odexej (mgr.14176) 1531 : cluster [DBG] pgmap v1052: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:48.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:49.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:48 smithi006 bash[26022]: audit 2023-11-17T06:06:47.272973+0000 mgr.smithi006.odexej (mgr.14176) 1532 : audit [DBG] from='client.15302 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:49.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:48 smithi156 bash[29655]: audit 2023-11-17T06:06:47.272973+0000 mgr.smithi006.odexej (mgr.14176) 1532 : audit [DBG] from='client.15302 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:50.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:49 smithi006 bash[26022]: cluster 2023-11-17T06:06:49.101815+0000 mgr.smithi006.odexej (mgr.14176) 1533 : cluster [DBG] pgmap v1053: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:50.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:49 smithi156 bash[29655]: cluster 2023-11-17T06:06:49.101815+0000 mgr.smithi006.odexej (mgr.14176) 1533 : cluster [DBG] pgmap v1053: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:51.877 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:51.877 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:52.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:52 smithi006 bash[26022]: cluster 2023-11-17T06:06:51.103210+0000 mgr.smithi006.odexej (mgr.14176) 1534 : cluster [DBG] pgmap v1054: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:52.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:52 smithi156 bash[29655]: cluster 2023-11-17T06:06:51.103210+0000 mgr.smithi006.odexej (mgr.14176) 1534 : cluster [DBG] pgmap v1054: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:52.448 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:53.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:53 smithi156 bash[29655]: audit 2023-11-17T06:06:51.862823+0000 mgr.smithi006.odexej (mgr.14176) 1535 : audit [DBG] from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:53.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:53.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:53 smithi006 bash[26022]: audit 2023-11-17T06:06:51.862823+0000 mgr.smithi006.odexej (mgr.14176) 1535 : audit [DBG] from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:54.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:54 smithi156 bash[29655]: cluster 2023-11-17T06:06:53.104289+0000 mgr.smithi006.odexej (mgr.14176) 1536 : cluster [DBG] pgmap v1055: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:54.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:54 smithi006 bash[26022]: cluster 2023-11-17T06:06:53.104289+0000 mgr.smithi006.odexej (mgr.14176) 1536 : cluster [DBG] pgmap v1055: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:06:55.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:55 smithi006 bash[26022]: audit 2023-11-17T06:06:55.074535+0000 mon.smithi006 (mon.0) 1044 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:06:55.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:55 smithi156 bash[29655]: audit 2023-11-17T06:06:55.074535+0000 mon.smithi006 (mon.0) 1044 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: cluster 2023-11-17T06:06:55.105889+0000 mgr.smithi006.odexej (mgr.14176) 1537 : cluster [DBG] pgmap v1056: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: audit 2023-11-17T06:06:55.388306+0000 mon.smithi006 (mon.0) 1045 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: audit 2023-11-17T06:06:55.389815+0000 mon.smithi006 (mon.0) 1046 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: audit 2023-11-17T06:06:55.398033+0000 mon.smithi006 (mon.0) 1047 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: audit 2023-11-17T06:06:55.407848+0000 mon.smithi006 (mon.0) 1048 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:06:56.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:56 smithi006 bash[26022]: audit 2023-11-17T06:06:55.422568+0000 mon.smithi006 (mon.0) 1049 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:56.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:06:56.559 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:04:46.897158Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.tgqlnx on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:06:56.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: cluster 2023-11-17T06:06:55.105889+0000 mgr.smithi006.odexej (mgr.14176) 1537 : cluster [DBG] pgmap v1056: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:06:56.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: audit 2023-11-17T06:06:55.388306+0000 mon.smithi006 (mon.0) 1045 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:06:56.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: audit 2023-11-17T06:06:55.389815+0000 mon.smithi006 (mon.0) 1046 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:06:56.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: audit 2023-11-17T06:06:55.398033+0000 mon.smithi006 (mon.0) 1047 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:56.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: audit 2023-11-17T06:06:55.407848+0000 mon.smithi006 (mon.0) 1048 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:06:56.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:56 smithi156 bash[29655]: audit 2023-11-17T06:06:55.422568+0000 mon.smithi006 (mon.0) 1049 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:06:57.167 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:06:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:57 smithi006 bash[26022]: cluster 2023-11-17T06:06:55.399088+0000 mgr.smithi006.odexej (mgr.14176) 1538 : cluster [DBG] pgmap v1057: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:06:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:57 smithi006 bash[26022]: cephadm 2023-11-17T06:06:55.426145+0000 mgr.smithi006.odexej (mgr.14176) 1539 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.loiish on smithi156 2023-11-17T06:06:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:57 smithi006 bash[26022]: cluster 2023-11-17T06:06:56.394998+0000 mon.smithi006 (mon.0) 1050 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:06:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:57 smithi006 bash[26022]: cluster 2023-11-17T06:06:56.395049+0000 mon.smithi006 (mon.0) 1051 : cluster [INF] Cluster is now healthy 2023-11-17T06:06:57.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:57 smithi156 bash[29655]: cluster 2023-11-17T06:06:55.399088+0000 mgr.smithi006.odexej (mgr.14176) 1538 : cluster [DBG] pgmap v1057: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:06:57.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:57 smithi156 bash[29655]: cephadm 2023-11-17T06:06:55.426145+0000 mgr.smithi006.odexej (mgr.14176) 1539 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.loiish on smithi156 2023-11-17T06:06:57.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:57 smithi156 bash[29655]: cluster 2023-11-17T06:06:56.394998+0000 mon.smithi006 (mon.0) 1050 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:06:57.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:57 smithi156 bash[29655]: cluster 2023-11-17T06:06:56.395049+0000 mon.smithi006 (mon.0) 1051 : cluster [INF] Cluster is now healthy 2023-11-17T06:06:58.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:06:58.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:58 smithi006 bash[26022]: audit 2023-11-17T06:06:56.543481+0000 mgr.smithi006.odexej (mgr.14176) 1540 : audit [DBG] from='client.15310 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:58.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:58 smithi156 bash[29655]: audit 2023-11-17T06:06:56.543481+0000 mgr.smithi006.odexej (mgr.14176) 1540 : audit [DBG] from='client.15310 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:06:59.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:06:59 smithi006 bash[26022]: cluster 2023-11-17T06:06:57.399913+0000 mgr.smithi006.odexej (mgr.14176) 1541 : cluster [DBG] pgmap v1058: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:06:59.483 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:06:59 smithi156 bash[29655]: cluster 2023-11-17T06:06:57.399913+0000 mgr.smithi006.odexej (mgr.14176) 1541 : cluster [DBG] pgmap v1058: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:00.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:00 smithi006 bash[26022]: audit 2023-11-17T06:06:59.510621+0000 mon.smithi006 (mon.0) 1052 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.loiish"}]: dispatch 2023-11-17T06:07:00.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:00 smithi156 bash[29655]: audit 2023-11-17T06:06:59.510621+0000 mon.smithi006 (mon.0) 1052 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.loiish"}]: dispatch 2023-11-17T06:07:01.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:01.266 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:51.318253Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.kgzbrn on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-kgzbrn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.kgzbrn\nDeploy daemon haproxy.nfs.foo.smithi156.kgzbrn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.063153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hhjtxc on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hhjtxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hhjtxc\nDeploy daemon haproxy.nfs.foo.smithi006.hhjtxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:05:55.065378Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.mogsrs on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: cluster 2023-11-17T06:06:59.401133+0000 mgr.smithi006.odexej (mgr.14176) 1542 : cluster [DBG] pgmap v1059: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: cephadm 2023-11-17T06:06:59.509442+0000 mgr.smithi006.odexej (mgr.14176) 1543 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: cephadm 2023-11-17T06:06:59.509859+0000 mgr.smithi006.odexej (mgr.14176) 1544 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: cephadm 2023-11-17T06:06:59.511982+0000 mgr.smithi006.odexej (mgr.14176) 1545 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.681 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:01.681 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.681 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.681 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.681 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.682 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.682 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.682 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.682 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:01 smithi006 bash[26022]: cephadm 2023-11-17T06:06:59.516992+0000 mgr.smithi006.odexej (mgr.14176) 1546 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.baassd on smithi006 2023-11-17T06:07:01.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: cluster 2023-11-17T06:06:59.401133+0000 mgr.smithi006.odexej (mgr.14176) 1542 : cluster [DBG] pgmap v1059: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:01.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: cephadm 2023-11-17T06:06:59.509442+0000 mgr.smithi006.odexej (mgr.14176) 1543 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:07:01.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: cephadm 2023-11-17T06:06:59.509859+0000 mgr.smithi006.odexej (mgr.14176) 1544 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: cephadm 2023-11-17T06:06:59.511982+0000 mgr.smithi006.odexej (mgr.14176) 1545 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.loiish ... 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.691 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:01.692 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:01.692 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:01.692 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:01.692 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:01 smithi156 bash[29655]: cephadm 2023-11-17T06:06:59.516992+0000 mgr.smithi006.odexej (mgr.14176) 1546 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.baassd on smithi006 2023-11-17T06:07:01.921 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:02.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:02 smithi006 bash[26022]: audit 2023-11-17T06:07:01.254760+0000 mgr.smithi006.odexej (mgr.14176) 1547 : audit [DBG] from='client.15314 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:02.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:02 smithi156 bash[29655]: audit 2023-11-17T06:07:01.254760+0000 mgr.smithi006.odexej (mgr.14176) 1547 : audit [DBG] from='client.15314 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:02.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:03.501 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:03 smithi006 bash[26022]: cluster 2023-11-17T06:07:01.401826+0000 mgr.smithi006.odexej (mgr.14176) 1548 : cluster [DBG] pgmap v1060: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:03.623 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:03 smithi156 bash[29655]: cluster 2023-11-17T06:07:01.401826+0000 mgr.smithi006.odexej (mgr.14176) 1548 : cluster [DBG] pgmap v1060: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:04.505 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:04 smithi006 bash[26022]: audit 2023-11-17T06:07:03.538355+0000 mon.smithi006 (mon.0) 1053 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.baassd"}]: dispatch 2023-11-17T06:07:04.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:04 smithi156 bash[29655]: audit 2023-11-17T06:07:03.538355+0000 mon.smithi006 (mon.0) 1053 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.baassd"}]: dispatch 2023-11-17T06:07:05.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cluster 2023-11-17T06:07:03.402788+0000 mgr.smithi006.odexej (mgr.14176) 1549 : cluster [DBG] pgmap v1061: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cephadm 2023-11-17T06:07:03.537053+0000 mgr.smithi006.odexej (mgr.14176) 1550 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:07:05.675 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.676 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cephadm 2023-11-17T06:07:03.537660+0000 mgr.smithi006.odexej (mgr.14176) 1551 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cephadm 2023-11-17T06:07:03.539841+0000 mgr.smithi006.odexej (mgr.14176) 1552 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:07:05.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cephadm 2023-11-17T06:07:03.544248+0000 mgr.smithi006.odexej (mgr.14176) 1553 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cephadm 2023-11-17T06:07:03.548034+0000 mgr.smithi006.odexej (mgr.14176) 1554 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cluster 2023-11-17T06:07:03.549000+0000 mgr.smithi006.odexej (mgr.14176) 1555 : cluster [DBG] pgmap v1062: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 121 B/s rd, 0 op/s 2023-11-17T06:07:05.678 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:05 smithi006 bash[26022]: cluster 2023-11-17T06:07:04.230330+0000 mon.smithi006 (mon.0) 1054 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cluster 2023-11-17T06:07:03.402788+0000 mgr.smithi006.odexej (mgr.14176) 1549 : cluster [DBG] pgmap v1061: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cephadm 2023-11-17T06:07:03.537053+0000 mgr.smithi006.odexej (mgr.14176) 1550 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.695 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cephadm 2023-11-17T06:07:03.537660+0000 mgr.smithi006.odexej (mgr.14176) 1551 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cephadm 2023-11-17T06:07:03.539841+0000 mgr.smithi006.odexej (mgr.14176) 1552 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.baassd ... 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:07:05.696 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:07:05.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cephadm 2023-11-17T06:07:03.544248+0000 mgr.smithi006.odexej (mgr.14176) 1553 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:07:05.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cephadm 2023-11-17T06:07:03.548034+0000 mgr.smithi006.odexej (mgr.14176) 1554 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:07:05.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cluster 2023-11-17T06:07:03.549000+0000 mgr.smithi006.odexej (mgr.14176) 1555 : cluster [DBG] pgmap v1062: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 121 B/s rd, 0 op/s 2023-11-17T06:07:05.697 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:05 smithi156 bash[29655]: cluster 2023-11-17T06:07:04.230330+0000 mon.smithi006 (mon.0) 1054 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:07:06.300 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:06.300 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:06.920 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:07.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:07 smithi006 bash[26022]: cluster 2023-11-17T06:07:05.550366+0000 mgr.smithi006.odexej (mgr.14176) 1556 : cluster [DBG] pgmap v1063: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:07.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:07 smithi006 bash[26022]: audit 2023-11-17T06:07:06.260566+0000 mon.smithi006 (mon.0) 1055 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:07:07.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:07 smithi156 bash[29655]: cluster 2023-11-17T06:07:05.550366+0000 mgr.smithi006.odexej (mgr.14176) 1556 : cluster [DBG] pgmap v1063: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:07.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:07 smithi156 bash[29655]: audit 2023-11-17T06:07:06.260566+0000 mon.smithi006 (mon.0) 1055 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:07:07.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:08.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:08 smithi006 bash[26022]: audit 2023-11-17T06:07:06.286242+0000 mgr.smithi006.odexej (mgr.14176) 1557 : audit [DBG] from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:08.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:08 smithi156 bash[29655]: audit 2023-11-17T06:07:06.286242+0000 mgr.smithi006.odexej (mgr.14176) 1557 : audit [DBG] from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:09.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:09 smithi006 bash[26022]: cluster 2023-11-17T06:07:07.551368+0000 mgr.smithi006.odexej (mgr.14176) 1558 : cluster [DBG] pgmap v1064: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:09.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:09 smithi156 bash[29655]: cluster 2023-11-17T06:07:07.551368+0000 mgr.smithi006.odexej (mgr.14176) 1558 : cluster [DBG] pgmap v1064: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:10.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:10 smithi006 bash[26022]: cluster 2023-11-17T06:07:09.552815+0000 mgr.smithi006.odexej (mgr.14176) 1559 : cluster [DBG] pgmap v1065: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:10.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:10 smithi156 bash[29655]: cluster 2023-11-17T06:07:09.552815+0000 mgr.smithi006.odexej (mgr.14176) 1559 : cluster [DBG] pgmap v1065: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:10.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:10.823 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:11.450 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:11.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:11 smithi006 bash[26022]: audit 2023-11-17T06:07:10.813597+0000 mgr.smithi006.odexej (mgr.14176) 1560 : audit [DBG] from='client.15322 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:11.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:11 smithi156 bash[29655]: audit 2023-11-17T06:07:10.813597+0000 mgr.smithi006.odexej (mgr.14176) 1560 : audit [DBG] from='client.15322 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:12.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:13.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:12 smithi006 bash[26022]: cluster 2023-11-17T06:07:11.553907+0000 mgr.smithi006.odexej (mgr.14176) 1561 : cluster [DBG] pgmap v1066: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:12 smithi156 bash[29655]: cluster 2023-11-17T06:07:11.553907+0000 mgr.smithi006.odexej (mgr.14176) 1561 : cluster [DBG] pgmap v1066: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:15.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:14 smithi006 bash[26022]: cluster 2023-11-17T06:07:13.555110+0000 mgr.smithi006.odexej (mgr.14176) 1562 : cluster [DBG] pgmap v1067: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:15.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:14 smithi156 bash[29655]: cluster 2023-11-17T06:07:13.555110+0000 mgr.smithi006.odexej (mgr.14176) 1562 : cluster [DBG] pgmap v1067: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:07:15.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:15.623 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:16.251 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:17.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:16 smithi006 bash[26022]: cluster 2023-11-17T06:07:15.556033+0000 mgr.smithi006.odexej (mgr.14176) 1563 : cluster [DBG] pgmap v1068: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:17.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:16 smithi006 bash[26022]: audit 2023-11-17T06:07:15.610992+0000 mgr.smithi006.odexej (mgr.14176) 1564 : audit [DBG] from='client.15326 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:17.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:16 smithi156 bash[29655]: cluster 2023-11-17T06:07:15.556033+0000 mgr.smithi006.odexej (mgr.14176) 1563 : cluster [DBG] pgmap v1068: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:17.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:16 smithi156 bash[29655]: audit 2023-11-17T06:07:15.610992+0000 mgr.smithi006.odexej (mgr.14176) 1564 : audit [DBG] from='client.15326 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:17.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:19.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:18 smithi006 bash[26022]: cluster 2023-11-17T06:07:17.557071+0000 mgr.smithi006.odexej (mgr.14176) 1565 : cluster [DBG] pgmap v1069: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:19.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:18 smithi156 bash[29655]: cluster 2023-11-17T06:07:17.557071+0000 mgr.smithi006.odexej (mgr.14176) 1565 : cluster [DBG] pgmap v1069: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:20.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:20.408 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:20.971 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:21.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:20 smithi006 bash[26022]: cluster 2023-11-17T06:07:19.558497+0000 mgr.smithi006.odexej (mgr.14176) 1566 : cluster [DBG] pgmap v1070: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:21.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:20 smithi156 bash[29655]: cluster 2023-11-17T06:07:19.558497+0000 mgr.smithi006.odexej (mgr.14176) 1566 : cluster [DBG] pgmap v1070: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:21.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:22.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:21 smithi006 bash[26022]: audit 2023-11-17T06:07:20.393694+0000 mgr.smithi006.odexej (mgr.14176) 1567 : audit [DBG] from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:22.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:21 smithi156 bash[29655]: audit 2023-11-17T06:07:20.393694+0000 mgr.smithi006.odexej (mgr.14176) 1567 : audit [DBG] from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:23.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:22 smithi006 bash[26022]: cluster 2023-11-17T06:07:21.559610+0000 mgr.smithi006.odexej (mgr.14176) 1568 : cluster [DBG] pgmap v1071: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:23.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:22 smithi156 bash[29655]: cluster 2023-11-17T06:07:21.559610+0000 mgr.smithi006.odexej (mgr.14176) 1568 : cluster [DBG] pgmap v1071: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:25.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:25.015 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:25.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:24 smithi006 bash[26022]: cluster 2023-11-17T06:07:23.560903+0000 mgr.smithi006.odexej (mgr.14176) 1569 : cluster [DBG] pgmap v1072: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:07:25.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:24 smithi156 bash[29655]: cluster 2023-11-17T06:07:23.560903+0000 mgr.smithi006.odexej (mgr.14176) 1569 : cluster [DBG] pgmap v1072: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:07:25.619 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:26.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:25 smithi006 bash[26022]: audit 2023-11-17T06:07:24.997928+0000 mgr.smithi006.odexej (mgr.14176) 1570 : audit [DBG] from='client.15334 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:26.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:25 smithi156 bash[29655]: audit 2023-11-17T06:07:24.997928+0000 mgr.smithi006.odexej (mgr.14176) 1570 : audit [DBG] from='client.15334 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:07:26.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:07:27.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:26 smithi006 bash[26022]: cluster 2023-11-17T06:07:25.562317+0000 mgr.smithi006.odexej (mgr.14176) 1571 : cluster [DBG] pgmap v1073: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:27.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:26 smithi156 bash[29655]: cluster 2023-11-17T06:07:25.562317+0000 mgr.smithi006.odexej (mgr.14176) 1571 : cluster [DBG] pgmap v1073: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:29.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:28 smithi006 bash[26022]: cluster 2023-11-17T06:07:27.563429+0000 mgr.smithi006.odexej (mgr.14176) 1572 : cluster [DBG] pgmap v1074: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:29.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:28 smithi156 bash[29655]: cluster 2023-11-17T06:07:27.563429+0000 mgr.smithi006.odexej (mgr.14176) 1572 : cluster [DBG] pgmap v1074: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:07:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:07:29.678 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:05:55.069040Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.xcrqtg on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:06:59.511851Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.loiish on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-loiish\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.loiish\nDeploy daemon haproxy.nfs.foo.smithi156.loiish ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.539555Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.baassd on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-baassd\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.baassd\nDeploy daemon haproxy.nfs.foo.smithi006.baassd ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:07:03.544002Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ameysu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:07:30.264 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:07:31.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:30 smithi156 bash[29655]: cluster 2023-11-17T06:07:29.564389+0000 mgr.smithi006.odexej (mgr.14176) 1573 : cluster [DBG] pgmap v1075: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:08.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:30 smithi156 bash[29655]: audit 2023-11-17T06:07:29.660297+0000 mgr.smithi006.odexej (mgr.14176) 1574 : audit [DBG] from='client.15338 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:08.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:08.902 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:32 smithi156 bash[29655]: cluster 2023-11-17T06:07:31.565337+0000 mgr.smithi006.odexej (mgr.14176) 1575 : cluster [DBG] pgmap v1076: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.088 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:34 smithi156 bash[29655]: cluster 2023-11-17T06:07:33.566540+0000 mgr.smithi006.odexej (mgr.14176) 1576 : cluster [DBG] pgmap v1077: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:09.088 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:36 smithi156 bash[29655]: cluster 2023-11-17T06:07:35.567906+0000 mgr.smithi006.odexej (mgr.14176) 1577 : cluster [DBG] pgmap v1078: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.091 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:38 smithi156 bash[29655]: cluster 2023-11-17T06:07:37.568911+0000 mgr.smithi006.odexej (mgr.14176) 1578 : cluster [DBG] pgmap v1079: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:40 smithi156 bash[29655]: cluster 2023-11-17T06:07:39.570278+0000 mgr.smithi006.odexej (mgr.14176) 1579 : cluster [DBG] pgmap v1080: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:42 smithi156 bash[29655]: cluster 2023-11-17T06:07:41.571320+0000 mgr.smithi006.odexej (mgr.14176) 1580 : cluster [DBG] pgmap v1081: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:44 smithi156 bash[29655]: cluster 2023-11-17T06:07:43.572512+0000 mgr.smithi006.odexej (mgr.14176) 1581 : cluster [DBG] pgmap v1082: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:46 smithi156 bash[29655]: cluster 2023-11-17T06:07:45.573942+0000 mgr.smithi006.odexej (mgr.14176) 1582 : cluster [DBG] pgmap v1083: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:48 smithi156 bash[29655]: cluster 2023-11-17T06:07:47.574978+0000 mgr.smithi006.odexej (mgr.14176) 1583 : cluster [DBG] pgmap v1084: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:50 smithi156 bash[29655]: cluster 2023-11-17T06:07:49.576366+0000 mgr.smithi006.odexej (mgr.14176) 1584 : cluster [DBG] pgmap v1085: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:52 smithi156 bash[29655]: cluster 2023-11-17T06:07:51.577404+0000 mgr.smithi006.odexej (mgr.14176) 1585 : cluster [DBG] pgmap v1086: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:54 smithi156 bash[29655]: cluster 2023-11-17T06:07:53.578655+0000 mgr.smithi006.odexej (mgr.14176) 1586 : cluster [DBG] pgmap v1087: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:56 smithi156 bash[29655]: cluster 2023-11-17T06:07:55.580020+0000 mgr.smithi006.odexej (mgr.14176) 1587 : cluster [DBG] pgmap v1088: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:07:59 smithi156 bash[29655]: cluster 2023-11-17T06:07:57.581039+0000 mgr.smithi006.odexej (mgr.14176) 1588 : cluster [DBG] pgmap v1089: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:01 smithi156 bash[29655]: cluster 2023-11-17T06:07:59.582526+0000 mgr.smithi006.odexej (mgr.14176) 1589 : cluster [DBG] pgmap v1090: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:03 smithi156 bash[29655]: cluster 2023-11-17T06:08:01.583628+0000 mgr.smithi006.odexej (mgr.14176) 1590 : cluster [DBG] pgmap v1091: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:09.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.552239+0000 mon.smithi006 (mon.0) 1056 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:08:09.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.865926+0000 mon.smithi006 (mon.0) 1057 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:08:09.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.867523+0000 mon.smithi006 (mon.0) 1058 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:08:09.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.876217+0000 mon.smithi006 (mon.0) 1059 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:09.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.884437+0000 mon.smithi006 (mon.0) 1060 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:04 smithi156 bash[29655]: audit 2023-11-17T06:08:03.897741+0000 mon.smithi006 (mon.0) 1061 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: cluster 2023-11-17T06:08:03.584748+0000 mgr.smithi006.odexej (mgr.14176) 1591 : cluster [DBG] pgmap v1092: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: cluster 2023-11-17T06:08:03.877034+0000 mgr.smithi006.odexej (mgr.14176) 1592 : cluster [DBG] pgmap v1093: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: cephadm 2023-11-17T06:08:03.901425+0000 mgr.smithi006.odexej (mgr.14176) 1593 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.tfumst on smithi156 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: cluster 2023-11-17T06:08:04.012542+0000 mon.smithi006 (mon.0) 1062 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: cluster 2023-11-17T06:08:04.012626+0000 mon.smithi006 (mon.0) 1063 : cluster [INF] Cluster is now healthy 2023-11-17T06:08:09.206 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:05 smithi156 bash[29655]: audit 2023-11-17T06:08:04.743101+0000 mon.smithi006 (mon.0) 1064 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:07 smithi156 bash[29655]: cluster 2023-11-17T06:08:05.878066+0000 mgr.smithi006.odexej (mgr.14176) 1594 : cluster [DBG] pgmap v1094: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:08 smithi156 bash[29655]: audit 2023-11-17T06:08:08.007766+0000 mon.smithi006 (mon.0) 1065 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.tfumst"}]: dispatch 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:30 smithi006 bash[26022]: cluster 2023-11-17T06:07:29.564389+0000 mgr.smithi006.odexej (mgr.14176) 1573 : cluster [DBG] pgmap v1075: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:30 smithi006 bash[26022]: audit 2023-11-17T06:07:29.660297+0000 mgr.smithi006.odexej (mgr.14176) 1574 : audit [DBG] from='client.15338 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:32 smithi006 bash[26022]: cluster 2023-11-17T06:07:31.565337+0000 mgr.smithi006.odexej (mgr.14176) 1575 : cluster [DBG] pgmap v1076: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.161 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:34 smithi006 bash[26022]: cluster 2023-11-17T06:07:33.566540+0000 mgr.smithi006.odexej (mgr.14176) 1576 : cluster [DBG] pgmap v1077: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:36 smithi006 bash[26022]: cluster 2023-11-17T06:07:35.567906+0000 mgr.smithi006.odexej (mgr.14176) 1577 : cluster [DBG] pgmap v1078: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:38 smithi006 bash[26022]: cluster 2023-11-17T06:07:37.568911+0000 mgr.smithi006.odexej (mgr.14176) 1578 : cluster [DBG] pgmap v1079: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:40 smithi006 bash[26022]: cluster 2023-11-17T06:07:39.570278+0000 mgr.smithi006.odexej (mgr.14176) 1579 : cluster [DBG] pgmap v1080: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:42 smithi006 bash[26022]: cluster 2023-11-17T06:07:41.571320+0000 mgr.smithi006.odexej (mgr.14176) 1580 : cluster [DBG] pgmap v1081: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:44 smithi006 bash[26022]: cluster 2023-11-17T06:07:43.572512+0000 mgr.smithi006.odexej (mgr.14176) 1581 : cluster [DBG] pgmap v1082: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:46 smithi006 bash[26022]: cluster 2023-11-17T06:07:45.573942+0000 mgr.smithi006.odexej (mgr.14176) 1582 : cluster [DBG] pgmap v1083: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:48 smithi006 bash[26022]: cluster 2023-11-17T06:07:47.574978+0000 mgr.smithi006.odexej (mgr.14176) 1583 : cluster [DBG] pgmap v1084: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:50 smithi006 bash[26022]: cluster 2023-11-17T06:07:49.576366+0000 mgr.smithi006.odexej (mgr.14176) 1584 : cluster [DBG] pgmap v1085: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:52 smithi006 bash[26022]: cluster 2023-11-17T06:07:51.577404+0000 mgr.smithi006.odexej (mgr.14176) 1585 : cluster [DBG] pgmap v1086: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:54 smithi006 bash[26022]: cluster 2023-11-17T06:07:53.578655+0000 mgr.smithi006.odexej (mgr.14176) 1586 : cluster [DBG] pgmap v1087: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:56 smithi006 bash[26022]: cluster 2023-11-17T06:07:55.580020+0000 mgr.smithi006.odexej (mgr.14176) 1587 : cluster [DBG] pgmap v1088: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.162 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:07:59 smithi006 bash[26022]: cluster 2023-11-17T06:07:57.581039+0000 mgr.smithi006.odexej (mgr.14176) 1588 : cluster [DBG] pgmap v1089: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:01 smithi006 bash[26022]: cluster 2023-11-17T06:07:59.582526+0000 mgr.smithi006.odexej (mgr.14176) 1589 : cluster [DBG] pgmap v1090: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:03 smithi006 bash[26022]: cluster 2023-11-17T06:08:01.583628+0000 mgr.smithi006.odexej (mgr.14176) 1590 : cluster [DBG] pgmap v1091: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.552239+0000 mon.smithi006 (mon.0) 1056 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.865926+0000 mon.smithi006 (mon.0) 1057 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.867523+0000 mon.smithi006 (mon.0) 1058 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.876217+0000 mon.smithi006 (mon.0) 1059 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.884437+0000 mon.smithi006 (mon.0) 1060 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:04 smithi006 bash[26022]: audit 2023-11-17T06:08:03.897741+0000 mon.smithi006 (mon.0) 1061 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: cluster 2023-11-17T06:08:03.584748+0000 mgr.smithi006.odexej (mgr.14176) 1591 : cluster [DBG] pgmap v1092: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: cluster 2023-11-17T06:08:03.877034+0000 mgr.smithi006.odexej (mgr.14176) 1592 : cluster [DBG] pgmap v1093: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: cephadm 2023-11-17T06:08:03.901425+0000 mgr.smithi006.odexej (mgr.14176) 1593 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.tfumst on smithi156 2023-11-17T06:08:10.163 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: cluster 2023-11-17T06:08:04.012542+0000 mon.smithi006 (mon.0) 1062 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:08:10.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: cluster 2023-11-17T06:08:04.012626+0000 mon.smithi006 (mon.0) 1063 : cluster [INF] Cluster is now healthy 2023-11-17T06:08:10.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:05 smithi006 bash[26022]: audit 2023-11-17T06:08:04.743101+0000 mon.smithi006 (mon.0) 1064 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:10.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:07 smithi006 bash[26022]: cluster 2023-11-17T06:08:05.878066+0000 mgr.smithi006.odexej (mgr.14176) 1594 : cluster [DBG] pgmap v1094: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:10.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:08 smithi006 bash[26022]: audit 2023-11-17T06:08:08.007766+0000 mon.smithi006 (mon.0) 1065 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.tfumst"}]: dispatch 2023-11-17T06:08:10.633 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: cluster 2023-11-17T06:08:07.878821+0000 mgr.smithi006.odexej (mgr.14176) 1595 : cluster [DBG] pgmap v1095: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:10.633 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: cephadm 2023-11-17T06:08:08.006499+0000 mgr.smithi006.odexej (mgr.14176) 1596 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:10.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: cephadm 2023-11-17T06:08:08.007007+0000 mgr.smithi006.odexej (mgr.14176) 1597 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: cephadm 2023-11-17T06:08:08.009665+0000 mgr.smithi006.odexej (mgr.14176) 1598 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.675 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.675 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.675 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:10.792 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:10.792 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.793 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:09 smithi006 bash[26022]: cephadm 2023-11-17T06:08:08.014724+0000 mgr.smithi006.odexej (mgr.14176) 1599 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.djzaro on smithi006 2023-11-17T06:08:10.793 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: cluster 2023-11-17T06:08:07.878821+0000 mgr.smithi006.odexej (mgr.14176) 1595 : cluster [DBG] pgmap v1095: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:10.793 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: cephadm 2023-11-17T06:08:08.006499+0000 mgr.smithi006.odexej (mgr.14176) 1596 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.793 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.793 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.794 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.794 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.795 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.795 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.795 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.795 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.921 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:10.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:10.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:08:10.922 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:08:10.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:08:10.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:08:10.923 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:08:10.924 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:10.969 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: cephadm 2023-11-17T06:08:08.007007+0000 mgr.smithi006.odexej (mgr.14176) 1597 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: cephadm 2023-11-17T06:08:08.009665+0000 mgr.smithi006.odexej (mgr.14176) 1598 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.tfumst ... 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:10.970 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:11.012 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:11.012 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:11.012 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:09 smithi156 bash[29655]: cephadm 2023-11-17T06:08:08.014724+0000 mgr.smithi006.odexej (mgr.14176) 1599 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.djzaro on smithi006 2023-11-17T06:08:11.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:11 smithi006 bash[26022]: cluster 2023-11-17T06:08:09.879989+0000 mgr.smithi006.odexej (mgr.14176) 1600 : cluster [DBG] pgmap v1096: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:11.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:11 smithi156 bash[29655]: cluster 2023-11-17T06:08:09.879989+0000 mgr.smithi006.odexej (mgr.14176) 1600 : cluster [DBG] pgmap v1096: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:13.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cluster 2023-11-17T06:08:11.880757+0000 mgr.smithi006.odexej (mgr.14176) 1601 : cluster [DBG] pgmap v1097: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:13.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cephadm 2023-11-17T06:08:11.895174+0000 mgr.smithi006.odexej (mgr.14176) 1602 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:08:13.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cephadm 2023-11-17T06:08:11.895416+0000 mgr.smithi006.odexej (mgr.14176) 1603 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: audit 2023-11-17T06:08:11.895963+0000 mon.smithi006 (mon.0) 1066 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.djzaro"}]: dispatch 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cephadm 2023-11-17T06:08:11.897472+0000 mgr.smithi006.odexej (mgr.14176) 1604 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cephadm 2023-11-17T06:08:11.899419+0000 mgr.smithi006.odexej (mgr.14176) 1605 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:08:13.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cephadm 2023-11-17T06:08:11.901036+0000 mgr.smithi006.odexej (mgr.14176) 1606 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:08:13.170 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:12 smithi006 bash[26022]: cluster 2023-11-17T06:08:11.902648+0000 mgr.smithi006.odexej (mgr.14176) 1607 : cluster [DBG] pgmap v1098: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:08:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cluster 2023-11-17T06:08:11.880757+0000 mgr.smithi006.odexej (mgr.14176) 1601 : cluster [DBG] pgmap v1097: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:08:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cephadm 2023-11-17T06:08:11.895174+0000 mgr.smithi006.odexej (mgr.14176) 1602 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:08:13.183 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.184 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cephadm 2023-11-17T06:08:11.895416+0000 mgr.smithi006.odexej (mgr.14176) 1603 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: audit 2023-11-17T06:08:11.895963+0000 mon.smithi006 (mon.0) 1066 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.djzaro"}]: dispatch 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cephadm 2023-11-17T06:08:11.897472+0000 mgr.smithi006.odexej (mgr.14176) 1604 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.djzaro ... 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.185 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cephadm 2023-11-17T06:08:11.899419+0000 mgr.smithi006.odexej (mgr.14176) 1605 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cephadm 2023-11-17T06:08:11.901036+0000 mgr.smithi006.odexej (mgr.14176) 1606 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:08:13.186 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:12 smithi156 bash[29655]: cluster 2023-11-17T06:08:11.902648+0000 mgr.smithi006.odexej (mgr.14176) 1607 : cluster [DBG] pgmap v1098: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:08:14.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:14.080 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:14.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:13 smithi006 bash[26022]: cluster 2023-11-17T06:08:12.863208+0000 mon.smithi006 (mon.0) 1067 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:08:14.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:13 smithi156 bash[29655]: cluster 2023-11-17T06:08:12.863208+0000 mon.smithi006 (mon.0) 1067 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:08:14.688 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:15.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:14 smithi006 bash[26022]: cluster 2023-11-17T06:08:13.903762+0000 mgr.smithi006.odexej (mgr.14176) 1608 : cluster [DBG] pgmap v1099: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:15.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:14 smithi006 bash[26022]: audit 2023-11-17T06:08:14.064127+0000 mgr.smithi006.odexej (mgr.14176) 1609 : audit [DBG] from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:15.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:14 smithi156 bash[29655]: cluster 2023-11-17T06:08:13.903762+0000 mgr.smithi006.odexej (mgr.14176) 1608 : cluster [DBG] pgmap v1099: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:15.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:14 smithi156 bash[29655]: audit 2023-11-17T06:08:14.064127+0000 mgr.smithi006.odexej (mgr.14176) 1609 : audit [DBG] from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:15.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:17.055 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:16 smithi006 bash[26022]: cluster 2023-11-17T06:08:15.904886+0000 mgr.smithi006.odexej (mgr.14176) 1610 : cluster [DBG] pgmap v1100: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:17.056 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:16 smithi006 bash[26022]: audit 2023-11-17T06:08:16.314037+0000 mon.smithi006 (mon.0) 1068 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:17.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:16 smithi156 bash[29655]: cluster 2023-11-17T06:08:15.904886+0000 mgr.smithi006.odexej (mgr.14176) 1610 : cluster [DBG] pgmap v1100: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:17.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:16 smithi156 bash[29655]: audit 2023-11-17T06:08:16.314037+0000 mon.smithi006 (mon.0) 1068 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:18.744 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:18.745 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:19.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:18 smithi006 bash[26022]: cluster 2023-11-17T06:08:17.905937+0000 mgr.smithi006.odexej (mgr.14176) 1611 : cluster [DBG] pgmap v1101: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:19.304 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:19.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:18 smithi156 bash[29655]: cluster 2023-11-17T06:08:17.905937+0000 mgr.smithi006.odexej (mgr.14176) 1611 : cluster [DBG] pgmap v1101: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:20.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:20.318 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:19 smithi006 bash[26022]: audit 2023-11-17T06:08:18.735845+0000 mgr.smithi006.odexej (mgr.14176) 1612 : audit [DBG] from='client.15346 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:20.318 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:19 smithi006 bash[26022]: audit 2023-11-17T06:08:19.745575+0000 mon.smithi006 (mon.0) 1069 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:20.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:19 smithi156 bash[29655]: audit 2023-11-17T06:08:18.735845+0000 mgr.smithi006.odexej (mgr.14176) 1612 : audit [DBG] from='client.15346 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:20.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:19 smithi156 bash[29655]: audit 2023-11-17T06:08:19.745575+0000 mon.smithi006 (mon.0) 1069 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:08:21.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:20 smithi006 bash[26022]: cluster 2023-11-17T06:08:19.907391+0000 mgr.smithi006.odexej (mgr.14176) 1613 : cluster [DBG] pgmap v1102: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:21.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:20 smithi156 bash[29655]: cluster 2023-11-17T06:08:19.907391+0000 mgr.smithi006.odexej (mgr.14176) 1613 : cluster [DBG] pgmap v1102: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:23.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:22 smithi006 bash[26022]: cluster 2023-11-17T06:08:21.908400+0000 mgr.smithi006.odexej (mgr.14176) 1614 : cluster [DBG] pgmap v1103: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:23.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:22 smithi156 bash[29655]: cluster 2023-11-17T06:08:21.908400+0000 mgr.smithi006.odexej (mgr.14176) 1614 : cluster [DBG] pgmap v1103: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2023-11-17T06:08:23.468 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:23.469 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:24.094 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:25.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:25.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:24 smithi006 bash[26022]: audit 2023-11-17T06:08:23.453290+0000 mgr.smithi006.odexej (mgr.14176) 1615 : audit [DBG] from='client.15350 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:24 smithi006 bash[26022]: cluster 2023-11-17T06:08:23.909587+0000 mgr.smithi006.odexej (mgr.14176) 1616 : cluster [DBG] pgmap v1104: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:25.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:24 smithi156 bash[29655]: audit 2023-11-17T06:08:23.453290+0000 mgr.smithi006.odexej (mgr.14176) 1615 : audit [DBG] from='client.15350 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:25.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:24 smithi156 bash[29655]: cluster 2023-11-17T06:08:23.909587+0000 mgr.smithi006.odexej (mgr.14176) 1616 : cluster [DBG] pgmap v1104: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:27.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:26 smithi006 bash[26022]: cluster 2023-11-17T06:08:25.910339+0000 mgr.smithi006.odexej (mgr.14176) 1617 : cluster [DBG] pgmap v1105: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:27.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:26 smithi156 bash[29655]: cluster 2023-11-17T06:08:25.910339+0000 mgr.smithi006.odexej (mgr.14176) 1617 : cluster [DBG] pgmap v1105: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:28.248 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:28.248 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:28.830 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:29.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:28 smithi006 bash[26022]: cluster 2023-11-17T06:08:27.911096+0000 mgr.smithi006.odexej (mgr.14176) 1618 : cluster [DBG] pgmap v1106: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:29.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:28 smithi006 bash[26022]: audit 2023-11-17T06:08:28.234910+0000 mgr.smithi006.odexej (mgr.14176) 1619 : audit [DBG] from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:29.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:28 smithi156 bash[29655]: cluster 2023-11-17T06:08:27.911096+0000 mgr.smithi006.odexej (mgr.14176) 1618 : cluster [DBG] pgmap v1106: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:29.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:28 smithi156 bash[29655]: audit 2023-11-17T06:08:28.234910+0000 mgr.smithi006.odexej (mgr.14176) 1619 : audit [DBG] from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:29.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:31.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:30 smithi006 bash[26022]: cluster 2023-11-17T06:08:29.912526+0000 mgr.smithi006.odexej (mgr.14176) 1620 : cluster [DBG] pgmap v1107: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:31.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:30 smithi156 bash[29655]: cluster 2023-11-17T06:08:29.912526+0000 mgr.smithi006.odexej (mgr.14176) 1620 : cluster [DBG] pgmap v1107: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:32.997 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:32.997 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:33.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:32 smithi006 bash[26022]: cluster 2023-11-17T06:08:31.913441+0000 mgr.smithi006.odexej (mgr.14176) 1621 : cluster [DBG] pgmap v1108: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:33.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:32 smithi156 bash[29655]: cluster 2023-11-17T06:08:31.913441+0000 mgr.smithi006.odexej (mgr.14176) 1621 : cluster [DBG] pgmap v1108: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:33.593 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:34.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:33 smithi006 bash[26022]: audit 2023-11-17T06:08:32.983432+0000 mgr.smithi006.odexej (mgr.14176) 1622 : audit [DBG] from='client.15358 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:34.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:33 smithi156 bash[29655]: audit 2023-11-17T06:08:32.983432+0000 mgr.smithi006.odexej (mgr.14176) 1622 : audit [DBG] from='client.15358 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:34.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:35.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:34 smithi006 bash[26022]: cluster 2023-11-17T06:08:33.914602+0000 mgr.smithi006.odexej (mgr.14176) 1623 : cluster [DBG] pgmap v1109: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:35.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:34 smithi156 bash[29655]: cluster 2023-11-17T06:08:33.914602+0000 mgr.smithi006.odexej (mgr.14176) 1623 : cluster [DBG] pgmap v1109: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:08:37.292 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:36 smithi006 bash[26022]: cluster 2023-11-17T06:08:35.915991+0000 mgr.smithi006.odexej (mgr.14176) 1624 : cluster [DBG] pgmap v1110: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:37.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:36 smithi156 bash[29655]: cluster 2023-11-17T06:08:35.915991+0000 mgr.smithi006.odexej (mgr.14176) 1624 : cluster [DBG] pgmap v1110: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:37.721 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:08:37.721 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:08:38.344 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:08:39.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:08:39.358 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:38 smithi006 bash[26022]: audit 2023-11-17T06:08:37.712289+0000 mgr.smithi006.odexej (mgr.14176) 1625 : audit [DBG] from='client.15362 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:39.358 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:38 smithi006 bash[26022]: cluster 2023-11-17T06:08:37.916798+0000 mgr.smithi006.odexej (mgr.14176) 1626 : cluster [DBG] pgmap v1111: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:39.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:38 smithi156 bash[29655]: audit 2023-11-17T06:08:37.712289+0000 mgr.smithi006.odexej (mgr.14176) 1625 : audit [DBG] from='client.15362 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:08:39.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:38 smithi156 bash[29655]: cluster 2023-11-17T06:08:37.916798+0000 mgr.smithi006.odexej (mgr.14176) 1626 : cluster [DBG] pgmap v1111: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:08:41.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:40 smithi006 bash[26022]: cluster 2023-11-17T06:08:39.918188+0000 mgr.smithi006.odexej (mgr.14176) 1627 : cluster [DBG] pgmap v1112: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.140 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:31.141 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:07:03.547917Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.osaxis on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:08.009402Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.tfumst on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-tfumst\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.tfumst\nDeploy daemon haproxy.nfs.foo.smithi156.tfumst ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.897357Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.djzaro on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-djzaro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.djzaro\nDeploy daemon haproxy.nfs.foo.smithi006.djzaro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:08:11.899315Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ljvtjy on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:31.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:42 smithi006 bash[26022]: cluster 2023-11-17T06:08:41.919063+0000 mgr.smithi006.odexej (mgr.14176) 1628 : cluster [DBG] pgmap v1113: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.191 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:40 smithi156 bash[29655]: cluster 2023-11-17T06:08:39.918188+0000 mgr.smithi006.odexej (mgr.14176) 1627 : cluster [DBG] pgmap v1112: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:42 smithi156 bash[29655]: cluster 2023-11-17T06:08:41.919063+0000 mgr.smithi006.odexej (mgr.14176) 1628 : cluster [DBG] pgmap v1113: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:43 smithi156 bash[29655]: audit 2023-11-17T06:08:42.570574+0000 mgr.smithi006.odexej (mgr.14176) 1629 : audit [DBG] from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:44 smithi156 bash[29655]: cluster 2023-11-17T06:08:43.920214+0000 mgr.smithi006.odexej (mgr.14176) 1630 : cluster [DBG] pgmap v1114: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:46 smithi156 bash[29655]: cluster 2023-11-17T06:08:45.921536+0000 mgr.smithi006.odexej (mgr.14176) 1631 : cluster [DBG] pgmap v1115: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:48 smithi156 bash[29655]: cluster 2023-11-17T06:08:47.922551+0000 mgr.smithi006.odexej (mgr.14176) 1632 : cluster [DBG] pgmap v1116: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:50 smithi156 bash[29655]: cluster 2023-11-17T06:08:49.923930+0000 mgr.smithi006.odexej (mgr.14176) 1633 : cluster [DBG] pgmap v1117: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:52 smithi156 bash[29655]: cluster 2023-11-17T06:08:51.924962+0000 mgr.smithi006.odexej (mgr.14176) 1634 : cluster [DBG] pgmap v1118: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:54 smithi156 bash[29655]: cluster 2023-11-17T06:08:53.926211+0000 mgr.smithi006.odexej (mgr.14176) 1635 : cluster [DBG] pgmap v1119: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.196 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:56 smithi156 bash[29655]: cluster 2023-11-17T06:08:55.927712+0000 mgr.smithi006.odexej (mgr.14176) 1636 : cluster [DBG] pgmap v1120: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:08:58 smithi156 bash[29655]: cluster 2023-11-17T06:08:57.928809+0000 mgr.smithi006.odexej (mgr.14176) 1637 : cluster [DBG] pgmap v1121: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:01 smithi156 bash[29655]: cluster 2023-11-17T06:08:59.930178+0000 mgr.smithi006.odexej (mgr.14176) 1638 : cluster [DBG] pgmap v1122: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:02 smithi156 bash[29655]: cluster 2023-11-17T06:09:01.931182+0000 mgr.smithi006.odexej (mgr.14176) 1639 : cluster [DBG] pgmap v1123: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:04 smithi156 bash[29655]: cluster 2023-11-17T06:09:03.932429+0000 mgr.smithi006.odexej (mgr.14176) 1640 : cluster [DBG] pgmap v1124: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:06 smithi156 bash[29655]: cluster 2023-11-17T06:09:05.933818+0000 mgr.smithi006.odexej (mgr.14176) 1641 : cluster [DBG] pgmap v1125: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:09 smithi156 bash[29655]: cluster 2023-11-17T06:09:07.934819+0000 mgr.smithi006.odexej (mgr.14176) 1642 : cluster [DBG] pgmap v1126: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:11 smithi156 bash[29655]: cluster 2023-11-17T06:09:09.936234+0000 mgr.smithi006.odexej (mgr.14176) 1643 : cluster [DBG] pgmap v1127: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:11.907602+0000 mon.smithi006 (mon.0) 1070 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: cluster 2023-11-17T06:09:11.937235+0000 mgr.smithi006.odexej (mgr.14176) 1644 : cluster [DBG] pgmap v1128: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:12.241004+0000 mon.smithi006 (mon.0) 1071 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:12.242839+0000 mon.smithi006 (mon.0) 1072 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:12.252895+0000 mon.smithi006 (mon.0) 1073 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: cluster 2023-11-17T06:09:12.253776+0000 mgr.smithi006.odexej (mgr.14176) 1645 : cluster [DBG] pgmap v1129: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.197 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:12.264603+0000 mon.smithi006 (mon.0) 1074 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: audit 2023-11-17T06:09:12.277181+0000 mon.smithi006 (mon.0) 1075 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:12 smithi156 bash[29655]: cephadm 2023-11-17T06:09:12.280640+0000 mgr.smithi006.odexej (mgr.14176) 1646 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.sagqyp on smithi156 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:13 smithi156 bash[29655]: cluster 2023-11-17T06:09:12.882361+0000 mon.smithi006 (mon.0) 1076 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:13 smithi156 bash[29655]: cluster 2023-11-17T06:09:12.882451+0000 mon.smithi006 (mon.0) 1077 : cluster [INF] Cluster is now healthy 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:14 smithi156 bash[29655]: cluster 2023-11-17T06:09:14.254785+0000 mgr.smithi006.odexej (mgr.14176) 1647 : cluster [DBG] pgmap v1130: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:16 smithi156 bash[29655]: audit 2023-11-17T06:09:15.979983+0000 mon.smithi006 (mon.0) 1078 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.sagqyp"}]: dispatch 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: cephadm 2023-11-17T06:09:15.978751+0000 mgr.smithi006.odexej (mgr.14176) 1648 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.198 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.199 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: cephadm 2023-11-17T06:09:15.979270+0000 mgr.smithi006.odexej (mgr.14176) 1649 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: cephadm 2023-11-17T06:09:15.981778+0000 mgr.smithi006.odexej (mgr.14176) 1650 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.200 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: cephadm 2023-11-17T06:09:15.986791+0000 mgr.smithi006.odexej (mgr.14176) 1651 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.msyjaj on smithi006 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:17 smithi156 bash[29655]: cluster 2023-11-17T06:09:16.255771+0000 mgr.smithi006.odexej (mgr.14176) 1652 : cluster [DBG] pgmap v1131: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:18 smithi156 bash[29655]: cluster 2023-11-17T06:09:18.256559+0000 mgr.smithi006.odexej (mgr.14176) 1653 : cluster [DBG] pgmap v1132: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cephadm 2023-11-17T06:09:19.746006+0000 mgr.smithi006.odexej (mgr.14176) 1654 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.201 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:09:31.202 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.203 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cephadm 2023-11-17T06:09:19.746506+0000 mgr.smithi006.odexej (mgr.14176) 1655 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: audit 2023-11-17T06:09:19.747214+0000 mon.smithi006 (mon.0) 1079 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: audit 2023-11-17T06:09:19.747458+0000 mon.smithi006 (mon.0) 1080 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.msyjaj"}]: dispatch 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cephadm 2023-11-17T06:09:19.749945+0000 mgr.smithi006.odexej (mgr.14176) 1656 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.204 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cephadm 2023-11-17T06:09:19.752168+0000 mgr.smithi006.odexej (mgr.14176) 1657 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cephadm 2023-11-17T06:09:19.753813+0000 mgr.smithi006.odexej (mgr.14176) 1658 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:20 smithi156 bash[29655]: cluster 2023-11-17T06:09:19.756066+0000 mgr.smithi006.odexej (mgr.14176) 1659 : cluster [DBG] pgmap v1133: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:21 smithi156 bash[29655]: cluster 2023-11-17T06:09:20.743401+0000 mon.smithi006 (mon.0) 1081 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:21 smithi156 bash[29655]: audit 2023-11-17T06:09:21.368200+0000 mon.smithi006 (mon.0) 1082 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:22 smithi156 bash[29655]: cluster 2023-11-17T06:09:21.757127+0000 mgr.smithi006.odexej (mgr.14176) 1660 : cluster [DBG] pgmap v1134: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:24 smithi156 bash[29655]: cluster 2023-11-17T06:09:23.758355+0000 mgr.smithi006.odexej (mgr.14176) 1661 : cluster [DBG] pgmap v1135: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:26 smithi156 bash[29655]: cluster 2023-11-17T06:09:25.759741+0000 mgr.smithi006.odexej (mgr.14176) 1662 : cluster [DBG] pgmap v1136: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:28 smithi156 bash[29655]: cluster 2023-11-17T06:09:27.760837+0000 mgr.smithi006.odexej (mgr.14176) 1663 : cluster [DBG] pgmap v1137: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:31.205 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:30 smithi156 bash[29655]: cluster 2023-11-17T06:09:29.762207+0000 mgr.smithi006.odexej (mgr.14176) 1664 : cluster [DBG] pgmap v1138: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:43 smithi006 bash[26022]: audit 2023-11-17T06:08:42.570574+0000 mgr.smithi006.odexej (mgr.14176) 1629 : audit [DBG] from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:44 smithi006 bash[26022]: cluster 2023-11-17T06:08:43.920214+0000 mgr.smithi006.odexej (mgr.14176) 1630 : cluster [DBG] pgmap v1114: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:46 smithi006 bash[26022]: cluster 2023-11-17T06:08:45.921536+0000 mgr.smithi006.odexej (mgr.14176) 1631 : cluster [DBG] pgmap v1115: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:48 smithi006 bash[26022]: cluster 2023-11-17T06:08:47.922551+0000 mgr.smithi006.odexej (mgr.14176) 1632 : cluster [DBG] pgmap v1116: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:50 smithi006 bash[26022]: cluster 2023-11-17T06:08:49.923930+0000 mgr.smithi006.odexej (mgr.14176) 1633 : cluster [DBG] pgmap v1117: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:52 smithi006 bash[26022]: cluster 2023-11-17T06:08:51.924962+0000 mgr.smithi006.odexej (mgr.14176) 1634 : cluster [DBG] pgmap v1118: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:54 smithi006 bash[26022]: cluster 2023-11-17T06:08:53.926211+0000 mgr.smithi006.odexej (mgr.14176) 1635 : cluster [DBG] pgmap v1119: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.206 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:56 smithi006 bash[26022]: cluster 2023-11-17T06:08:55.927712+0000 mgr.smithi006.odexej (mgr.14176) 1636 : cluster [DBG] pgmap v1120: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:08:58 smithi006 bash[26022]: cluster 2023-11-17T06:08:57.928809+0000 mgr.smithi006.odexej (mgr.14176) 1637 : cluster [DBG] pgmap v1121: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:01 smithi006 bash[26022]: cluster 2023-11-17T06:08:59.930178+0000 mgr.smithi006.odexej (mgr.14176) 1638 : cluster [DBG] pgmap v1122: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:02 smithi006 bash[26022]: cluster 2023-11-17T06:09:01.931182+0000 mgr.smithi006.odexej (mgr.14176) 1639 : cluster [DBG] pgmap v1123: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:04 smithi006 bash[26022]: cluster 2023-11-17T06:09:03.932429+0000 mgr.smithi006.odexej (mgr.14176) 1640 : cluster [DBG] pgmap v1124: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:06 smithi006 bash[26022]: cluster 2023-11-17T06:09:05.933818+0000 mgr.smithi006.odexej (mgr.14176) 1641 : cluster [DBG] pgmap v1125: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:09 smithi006 bash[26022]: cluster 2023-11-17T06:09:07.934819+0000 mgr.smithi006.odexej (mgr.14176) 1642 : cluster [DBG] pgmap v1126: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:11 smithi006 bash[26022]: cluster 2023-11-17T06:09:09.936234+0000 mgr.smithi006.odexej (mgr.14176) 1643 : cluster [DBG] pgmap v1127: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:11.907602+0000 mon.smithi006 (mon.0) 1070 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: cluster 2023-11-17T06:09:11.937235+0000 mgr.smithi006.odexej (mgr.14176) 1644 : cluster [DBG] pgmap v1128: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:12.241004+0000 mon.smithi006 (mon.0) 1071 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:12.242839+0000 mon.smithi006 (mon.0) 1072 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:12.252895+0000 mon.smithi006 (mon.0) 1073 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.207 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: cluster 2023-11-17T06:09:12.253776+0000 mgr.smithi006.odexej (mgr.14176) 1645 : cluster [DBG] pgmap v1129: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:12.264603+0000 mon.smithi006 (mon.0) 1074 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: audit 2023-11-17T06:09:12.277181+0000 mon.smithi006 (mon.0) 1075 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:12 smithi006 bash[26022]: cephadm 2023-11-17T06:09:12.280640+0000 mgr.smithi006.odexej (mgr.14176) 1646 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.sagqyp on smithi156 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:13 smithi006 bash[26022]: cluster 2023-11-17T06:09:12.882361+0000 mon.smithi006 (mon.0) 1076 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:13 smithi006 bash[26022]: cluster 2023-11-17T06:09:12.882451+0000 mon.smithi006 (mon.0) 1077 : cluster [INF] Cluster is now healthy 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:14 smithi006 bash[26022]: cluster 2023-11-17T06:09:14.254785+0000 mgr.smithi006.odexej (mgr.14176) 1647 : cluster [DBG] pgmap v1130: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:16 smithi006 bash[26022]: audit 2023-11-17T06:09:15.979983+0000 mon.smithi006 (mon.0) 1078 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.sagqyp"}]: dispatch 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: cephadm 2023-11-17T06:09:15.978751+0000 mgr.smithi006.odexej (mgr.14176) 1648 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.208 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.209 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: cephadm 2023-11-17T06:09:15.979270+0000 mgr.smithi006.odexej (mgr.14176) 1649 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: cephadm 2023-11-17T06:09:15.981778+0000 mgr.smithi006.odexej (mgr.14176) 1650 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.210 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.sagqyp ... 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: cephadm 2023-11-17T06:09:15.986791+0000 mgr.smithi006.odexej (mgr.14176) 1651 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.msyjaj on smithi006 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:17 smithi006 bash[26022]: cluster 2023-11-17T06:09:16.255771+0000 mgr.smithi006.odexej (mgr.14176) 1652 : cluster [DBG] pgmap v1131: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:18 smithi006 bash[26022]: cluster 2023-11-17T06:09:18.256559+0000 mgr.smithi006.odexej (mgr.14176) 1653 : cluster [DBG] pgmap v1132: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:09:31.211 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cephadm 2023-11-17T06:09:19.746006+0000 mgr.smithi006.odexej (mgr.14176) 1654 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:09:31.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.213 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cephadm 2023-11-17T06:09:19.746506+0000 mgr.smithi006.odexej (mgr.14176) 1655 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: audit 2023-11-17T06:09:19.747214+0000 mon.smithi006 (mon.0) 1079 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: audit 2023-11-17T06:09:19.747458+0000 mon.smithi006 (mon.0) 1080 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.msyjaj"}]: dispatch 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cephadm 2023-11-17T06:09:19.749945+0000 mgr.smithi006.odexej (mgr.14176) 1656 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.msyjaj ... 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cephadm 2023-11-17T06:09:19.752168+0000 mgr.smithi006.odexej (mgr.14176) 1657 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cephadm 2023-11-17T06:09:19.753813+0000 mgr.smithi006.odexej (mgr.14176) 1658 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:20 smithi006 bash[26022]: cluster 2023-11-17T06:09:19.756066+0000 mgr.smithi006.odexej (mgr.14176) 1659 : cluster [DBG] pgmap v1133: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:21 smithi006 bash[26022]: cluster 2023-11-17T06:09:20.743401+0000 mon.smithi006 (mon.0) 1081 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:21 smithi006 bash[26022]: audit 2023-11-17T06:09:21.368200+0000 mon.smithi006 (mon.0) 1082 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:22 smithi006 bash[26022]: cluster 2023-11-17T06:09:21.757127+0000 mgr.smithi006.odexej (mgr.14176) 1660 : cluster [DBG] pgmap v1134: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:24 smithi006 bash[26022]: cluster 2023-11-17T06:09:23.758355+0000 mgr.smithi006.odexej (mgr.14176) 1661 : cluster [DBG] pgmap v1135: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:26 smithi006 bash[26022]: cluster 2023-11-17T06:09:25.759741+0000 mgr.smithi006.odexej (mgr.14176) 1662 : cluster [DBG] pgmap v1136: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:31.215 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:28 smithi006 bash[26022]: cluster 2023-11-17T06:09:27.760837+0000 mgr.smithi006.odexej (mgr.14176) 1663 : cluster [DBG] pgmap v1137: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:31.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:30 smithi006 bash[26022]: cluster 2023-11-17T06:09:29.762207+0000 mgr.smithi006.odexej (mgr.14176) 1664 : cluster [DBG] pgmap v1138: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:09:32.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:33.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:32 smithi156 bash[29655]: cluster 2023-11-17T06:09:31.763176+0000 mgr.smithi006.odexej (mgr.14176) 1665 : cluster [DBG] pgmap v1139: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:33.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:32 smithi006 bash[26022]: cluster 2023-11-17T06:09:31.763176+0000 mgr.smithi006.odexej (mgr.14176) 1665 : cluster [DBG] pgmap v1139: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:35.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:34 smithi156 bash[29655]: cluster 2023-11-17T06:09:33.764125+0000 mgr.smithi006.odexej (mgr.14176) 1666 : cluster [DBG] pgmap v1140: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:35.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:34 smithi156 bash[29655]: audit 2023-11-17T06:09:34.748248+0000 mon.smithi006 (mon.0) 1083 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:35.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:34 smithi006 bash[26022]: cluster 2023-11-17T06:09:33.764125+0000 mgr.smithi006.odexej (mgr.14176) 1666 : cluster [DBG] pgmap v1140: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:35.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:34 smithi006 bash[26022]: audit 2023-11-17T06:09:34.748248+0000 mon.smithi006 (mon.0) 1083 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:09:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:35.491 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:36.124 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:37.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:37.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:36 smithi006 bash[26022]: audit 2023-11-17T06:09:35.482038+0000 mgr.smithi006.odexej (mgr.14176) 1667 : audit [DBG] from='client.15370 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:37.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:36 smithi006 bash[26022]: cluster 2023-11-17T06:09:35.765477+0000 mgr.smithi006.odexej (mgr.14176) 1668 : cluster [DBG] pgmap v1141: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:37.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:36 smithi156 bash[29655]: audit 2023-11-17T06:09:35.482038+0000 mgr.smithi006.odexej (mgr.14176) 1667 : audit [DBG] from='client.15370 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:37.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:36 smithi156 bash[29655]: cluster 2023-11-17T06:09:35.765477+0000 mgr.smithi006.odexej (mgr.14176) 1668 : cluster [DBG] pgmap v1141: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:39.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:38 smithi006 bash[26022]: cluster 2023-11-17T06:09:37.766531+0000 mgr.smithi006.odexej (mgr.14176) 1669 : cluster [DBG] pgmap v1142: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:39.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:38 smithi156 bash[29655]: cluster 2023-11-17T06:09:37.766531+0000 mgr.smithi006.odexej (mgr.14176) 1669 : cluster [DBG] pgmap v1142: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:40.130 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:40.130 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:40.739 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:41.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:40 smithi006 bash[26022]: cluster 2023-11-17T06:09:39.767821+0000 mgr.smithi006.odexej (mgr.14176) 1670 : cluster [DBG] pgmap v1143: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:41.169 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:40 smithi006 bash[26022]: audit 2023-11-17T06:09:40.113098+0000 mgr.smithi006.odexej (mgr.14176) 1671 : audit [DBG] from='client.15374 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:41.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:40 smithi156 bash[29655]: cluster 2023-11-17T06:09:39.767821+0000 mgr.smithi006.odexej (mgr.14176) 1670 : cluster [DBG] pgmap v1143: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:41.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:40 smithi156 bash[29655]: audit 2023-11-17T06:09:40.113098+0000 mgr.smithi006.odexej (mgr.14176) 1671 : audit [DBG] from='client.15374 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:41.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:43.038 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:42 smithi006 bash[26022]: cluster 2023-11-17T06:09:41.768442+0000 mgr.smithi006.odexej (mgr.14176) 1672 : cluster [DBG] pgmap v1144: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:43.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:42 smithi156 bash[29655]: cluster 2023-11-17T06:09:41.768442+0000 mgr.smithi006.odexej (mgr.14176) 1672 : cluster [DBG] pgmap v1144: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:44.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:44.850 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:45.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:44 smithi006 bash[26022]: cluster 2023-11-17T06:09:43.769695+0000 mgr.smithi006.odexej (mgr.14176) 1673 : cluster [DBG] pgmap v1145: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:45.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:44 smithi156 bash[29655]: cluster 2023-11-17T06:09:43.769695+0000 mgr.smithi006.odexej (mgr.14176) 1673 : cluster [DBG] pgmap v1145: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:45.479 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:46.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:45 smithi006 bash[26022]: audit 2023-11-17T06:09:44.832570+0000 mgr.smithi006.odexej (mgr.14176) 1674 : audit [DBG] from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:46.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:45 smithi156 bash[29655]: audit 2023-11-17T06:09:44.832570+0000 mgr.smithi006.odexej (mgr.14176) 1674 : audit [DBG] from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:46.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:47.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:46 smithi006 bash[26022]: cluster 2023-11-17T06:09:45.771196+0000 mgr.smithi006.odexej (mgr.14176) 1675 : cluster [DBG] pgmap v1146: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:47.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:46 smithi156 bash[29655]: cluster 2023-11-17T06:09:45.771196+0000 mgr.smithi006.odexej (mgr.14176) 1675 : cluster [DBG] pgmap v1146: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:49.239 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:48 smithi006 bash[26022]: cluster 2023-11-17T06:09:47.772311+0000 mgr.smithi006.odexej (mgr.14176) 1676 : cluster [DBG] pgmap v1147: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:50.219 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:48 smithi156 bash[29655]: cluster 2023-11-17T06:09:47.772311+0000 mgr.smithi006.odexej (mgr.14176) 1676 : cluster [DBG] pgmap v1147: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:50.220 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:50.220 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:50.220 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:51.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:51.234 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:50 smithi006 bash[26022]: audit 2023-11-17T06:09:49.588015+0000 mgr.smithi006.odexej (mgr.14176) 1677 : audit [DBG] from='client.15382 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:51.234 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:50 smithi006 bash[26022]: cluster 2023-11-17T06:09:49.773739+0000 mgr.smithi006.odexej (mgr.14176) 1678 : cluster [DBG] pgmap v1148: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:51.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:50 smithi156 bash[29655]: audit 2023-11-17T06:09:49.588015+0000 mgr.smithi006.odexej (mgr.14176) 1677 : audit [DBG] from='client.15382 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:51.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:50 smithi156 bash[29655]: cluster 2023-11-17T06:09:49.773739+0000 mgr.smithi006.odexej (mgr.14176) 1678 : cluster [DBG] pgmap v1148: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:53.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:52 smithi006 bash[26022]: cluster 2023-11-17T06:09:51.774788+0000 mgr.smithi006.odexej (mgr.14176) 1679 : cluster [DBG] pgmap v1149: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:53.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:52 smithi156 bash[29655]: cluster 2023-11-17T06:09:51.774788+0000 mgr.smithi006.odexej (mgr.14176) 1679 : cluster [DBG] pgmap v1149: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:54.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:54.558 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:55.164 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:09:55.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:54 smithi006 bash[26022]: cluster 2023-11-17T06:09:53.776043+0000 mgr.smithi006.odexej (mgr.14176) 1680 : cluster [DBG] pgmap v1150: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:55.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:54 smithi156 bash[29655]: cluster 2023-11-17T06:09:53.776043+0000 mgr.smithi006.odexej (mgr.14176) 1680 : cluster [DBG] pgmap v1150: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:09:56.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:09:56.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:55 smithi006 bash[26022]: audit 2023-11-17T06:09:54.545114+0000 mgr.smithi006.odexej (mgr.14176) 1681 : audit [DBG] from='client.15386 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:56.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:55 smithi156 bash[29655]: audit 2023-11-17T06:09:54.545114+0000 mgr.smithi006.odexej (mgr.14176) 1681 : audit [DBG] from='client.15386 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:09:57.289 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:57 smithi006 bash[26022]: cluster 2023-11-17T06:09:55.777395+0000 mgr.smithi006.odexej (mgr.14176) 1682 : cluster [DBG] pgmap v1151: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:57.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:57 smithi156 bash[29655]: cluster 2023-11-17T06:09:55.777395+0000 mgr.smithi006.odexej (mgr.14176) 1682 : cluster [DBG] pgmap v1151: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:59.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:09:59 smithi006 bash[26022]: cluster 2023-11-17T06:09:57.778531+0000 mgr.smithi006.odexej (mgr.14176) 1683 : cluster [DBG] pgmap v1152: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:59.231 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:09:59.231 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:09:59.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:09:59 smithi156 bash[29655]: cluster 2023-11-17T06:09:57.778531+0000 mgr.smithi006.odexej (mgr.14176) 1683 : cluster [DBG] pgmap v1152: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:09:59.852 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:00.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:00 smithi006 bash[26022]: audit 2023-11-17T06:09:59.222574+0000 mgr.smithi006.odexej (mgr.14176) 1684 : audit [DBG] from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:00.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:00 smithi006 bash[26022]: cluster 2023-11-17T06:10:00.000170+0000 mon.smithi006 (mon.0) 1084 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T06:10:00.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:00 smithi156 bash[29655]: audit 2023-11-17T06:09:59.222574+0000 mgr.smithi006.odexej (mgr.14176) 1684 : audit [DBG] from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:00.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:00 smithi156 bash[29655]: cluster 2023-11-17T06:10:00.000170+0000 mon.smithi006 (mon.0) 1084 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T06:10:00.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:01.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:01 smithi006 bash[26022]: cluster 2023-11-17T06:09:59.780026+0000 mgr.smithi006.odexej (mgr.14176) 1685 : cluster [DBG] pgmap v1153: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:03.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:03 smithi006 bash[26022]: cluster 2023-11-17T06:10:01.781156+0000 mgr.smithi006.odexej (mgr.14176) 1686 : cluster [DBG] pgmap v1154: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:03.916 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:01 smithi156 bash[29655]: cluster 2023-11-17T06:09:59.780026+0000 mgr.smithi006.odexej (mgr.14176) 1685 : cluster [DBG] pgmap v1153: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:03.916 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:03 smithi156 bash[29655]: cluster 2023-11-17T06:10:01.781156+0000 mgr.smithi006.odexej (mgr.14176) 1686 : cluster [DBG] pgmap v1154: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:03.947 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:03.947 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:04.382 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:05.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:05.396 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:05 smithi006 bash[26022]: audit 2023-11-17T06:10:03.746641+0000 mgr.smithi006.odexej (mgr.14176) 1687 : audit [DBG] from='client.15394 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:05.396 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:05 smithi006 bash[26022]: cluster 2023-11-17T06:10:03.781893+0000 mgr.smithi006.odexej (mgr.14176) 1688 : cluster [DBG] pgmap v1155: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:05.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:05 smithi156 bash[29655]: audit 2023-11-17T06:10:03.746641+0000 mgr.smithi006.odexej (mgr.14176) 1687 : audit [DBG] from='client.15394 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:05.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:05 smithi156 bash[29655]: cluster 2023-11-17T06:10:03.781893+0000 mgr.smithi006.odexej (mgr.14176) 1688 : cluster [DBG] pgmap v1155: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:07.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:07 smithi006 bash[26022]: cluster 2023-11-17T06:10:05.783222+0000 mgr.smithi006.odexej (mgr.14176) 1689 : cluster [DBG] pgmap v1156: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:07.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:07 smithi156 bash[29655]: cluster 2023-11-17T06:10:05.783222+0000 mgr.smithi006.odexej (mgr.14176) 1689 : cluster [DBG] pgmap v1156: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:08.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:08.547 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:09.183 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:09.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:09 smithi006 bash[26022]: cluster 2023-11-17T06:10:07.784382+0000 mgr.smithi006.odexej (mgr.14176) 1690 : cluster [DBG] pgmap v1157: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:09.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:09 smithi156 bash[29655]: cluster 2023-11-17T06:10:07.784382+0000 mgr.smithi006.odexej (mgr.14176) 1690 : cluster [DBG] pgmap v1157: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:10.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:10.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:10 smithi006 bash[26022]: audit 2023-11-17T06:10:08.529937+0000 mgr.smithi006.odexej (mgr.14176) 1691 : audit [DBG] from='client.15398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:10.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:10 smithi156 bash[29655]: audit 2023-11-17T06:10:08.529937+0000 mgr.smithi006.odexej (mgr.14176) 1691 : audit [DBG] from='client.15398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:11.351 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:11 smithi006 bash[26022]: cluster 2023-11-17T06:10:09.785845+0000 mgr.smithi006.odexej (mgr.14176) 1692 : cluster [DBG] pgmap v1158: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:11.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:11 smithi156 bash[29655]: cluster 2023-11-17T06:10:09.785845+0000 mgr.smithi006.odexej (mgr.14176) 1692 : cluster [DBG] pgmap v1158: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:13.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:13.265 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:13.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:13 smithi006 bash[26022]: cluster 2023-11-17T06:10:11.786985+0000 mgr.smithi006.odexej (mgr.14176) 1693 : cluster [DBG] pgmap v1159: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:13.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:13 smithi156 bash[29655]: cluster 2023-11-17T06:10:11.786985+0000 mgr.smithi006.odexej (mgr.14176) 1693 : cluster [DBG] pgmap v1159: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:13.891 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:14.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:14 smithi006 bash[26022]: audit 2023-11-17T06:10:13.245423+0000 mgr.smithi006.odexej (mgr.14176) 1694 : audit [DBG] from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:14.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:14 smithi156 bash[29655]: audit 2023-11-17T06:10:13.245423+0000 mgr.smithi006.odexej (mgr.14176) 1694 : audit [DBG] from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:14.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:15.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:15 smithi006 bash[26022]: cluster 2023-11-17T06:10:13.788250+0000 mgr.smithi006.odexej (mgr.14176) 1695 : cluster [DBG] pgmap v1160: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:15.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:15 smithi156 bash[29655]: cluster 2023-11-17T06:10:13.788250+0000 mgr.smithi006.odexej (mgr.14176) 1695 : cluster [DBG] pgmap v1160: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:17.412 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:17 smithi006 bash[26022]: cluster 2023-11-17T06:10:15.789667+0000 mgr.smithi006.odexej (mgr.14176) 1696 : cluster [DBG] pgmap v1161: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:17.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:17 smithi156 bash[29655]: cluster 2023-11-17T06:10:15.789667+0000 mgr.smithi006.odexej (mgr.14176) 1696 : cluster [DBG] pgmap v1161: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:17.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:17.867 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:18.569 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:19.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:19 smithi006 bash[26022]: cluster 2023-11-17T06:10:17.790696+0000 mgr.smithi006.odexej (mgr.14176) 1697 : cluster [DBG] pgmap v1162: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:19.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:19 smithi006 bash[26022]: audit 2023-11-17T06:10:17.851342+0000 mgr.smithi006.odexej (mgr.14176) 1698 : audit [DBG] from='client.15406 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:19.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:19 smithi156 bash[29655]: cluster 2023-11-17T06:10:17.790696+0000 mgr.smithi006.odexej (mgr.14176) 1697 : cluster [DBG] pgmap v1162: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:19.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:19 smithi156 bash[29655]: audit 2023-11-17T06:10:17.851342+0000 mgr.smithi006.odexej (mgr.14176) 1698 : audit [DBG] from='client.15406 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:19.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:20.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:20 smithi006 bash[26022]: audit 2023-11-17T06:10:19.763317+0000 mon.smithi006 (mon.0) 1085 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:10:20.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:20 smithi156 bash[29655]: audit 2023-11-17T06:10:19.763317+0000 mon.smithi006 (mon.0) 1085 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: cluster 2023-11-17T06:10:19.791630+0000 mgr.smithi006.odexej (mgr.14176) 1699 : cluster [DBG] pgmap v1163: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: audit 2023-11-17T06:10:20.092692+0000 mon.smithi006 (mon.0) 1086 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: audit 2023-11-17T06:10:20.094385+0000 mon.smithi006 (mon.0) 1087 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: audit 2023-11-17T06:10:20.106482+0000 mon.smithi006 (mon.0) 1088 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: cluster 2023-11-17T06:10:20.107184+0000 mgr.smithi006.odexej (mgr.14176) 1700 : cluster [DBG] pgmap v1164: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: audit 2023-11-17T06:10:20.112401+0000 mon.smithi006 (mon.0) 1089 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:10:21.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: audit 2023-11-17T06:10:20.127138+0000 mon.smithi006 (mon.0) 1090 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:21.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:21 smithi006 bash[26022]: cephadm 2023-11-17T06:10:20.132195+0000 mgr.smithi006.odexej (mgr.14176) 1701 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.izkwpl on smithi156 2023-11-17T06:10:21.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: cluster 2023-11-17T06:10:19.791630+0000 mgr.smithi006.odexej (mgr.14176) 1699 : cluster [DBG] pgmap v1163: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:21.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: audit 2023-11-17T06:10:20.092692+0000 mon.smithi006 (mon.0) 1086 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: audit 2023-11-17T06:10:20.094385+0000 mon.smithi006 (mon.0) 1087 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: audit 2023-11-17T06:10:20.106482+0000 mon.smithi006 (mon.0) 1088 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: cluster 2023-11-17T06:10:20.107184+0000 mgr.smithi006.odexej (mgr.14176) 1700 : cluster [DBG] pgmap v1164: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: audit 2023-11-17T06:10:20.112401+0000 mon.smithi006 (mon.0) 1089 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: audit 2023-11-17T06:10:20.127138+0000 mon.smithi006 (mon.0) 1090 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:21.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:21 smithi156 bash[29655]: cephadm 2023-11-17T06:10:20.132195+0000 mgr.smithi006.odexej (mgr.14176) 1701 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.izkwpl on smithi156 2023-11-17T06:10:22.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:22 smithi006 bash[26022]: cluster 2023-11-17T06:10:21.104957+0000 mon.smithi006 (mon.0) 1091 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:10:22.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:22 smithi006 bash[26022]: cluster 2023-11-17T06:10:21.105041+0000 mon.smithi006 (mon.0) 1092 : cluster [INF] Cluster is now healthy 2023-11-17T06:10:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:22 smithi156 bash[29655]: cluster 2023-11-17T06:10:21.104957+0000 mon.smithi006 (mon.0) 1091 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:10:22.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:22 smithi156 bash[29655]: cluster 2023-11-17T06:10:21.105041+0000 mon.smithi006 (mon.0) 1092 : cluster [INF] Cluster is now healthy 2023-11-17T06:10:22.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:22.660 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:08:11.900930Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.gkiijr on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:23.281 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:23.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:23 smithi006 bash[26022]: cluster 2023-11-17T06:10:22.107891+0000 mgr.smithi006.odexej (mgr.14176) 1702 : cluster [DBG] pgmap v1165: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:23.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:23 smithi156 bash[29655]: cluster 2023-11-17T06:10:22.107891+0000 mgr.smithi006.odexej (mgr.14176) 1702 : cluster [DBG] pgmap v1165: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:24.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:24.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:24 smithi006 bash[26022]: audit 2023-11-17T06:10:22.644068+0000 mgr.smithi006.odexej (mgr.14176) 1703 : audit [DBG] from='client.15410 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:24.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:24 smithi006 bash[26022]: audit 2023-11-17T06:10:23.842746+0000 mon.smithi006 (mon.0) 1093 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.izkwpl"}]: dispatch 2023-11-17T06:10:24.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:24 smithi156 bash[29655]: audit 2023-11-17T06:10:22.644068+0000 mgr.smithi006.odexej (mgr.14176) 1703 : audit [DBG] from='client.15410 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:24.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:24 smithi156 bash[29655]: audit 2023-11-17T06:10:23.842746+0000 mon.smithi006 (mon.0) 1093 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.izkwpl"}]: dispatch 2023-11-17T06:10:25.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: cephadm 2023-11-17T06:10:23.841565+0000 mgr.smithi006.odexej (mgr.14176) 1704 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:10:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:10:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:10:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:10:25.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:10:25.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.425 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.426 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.426 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.429 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.429 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.429 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.429 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: cephadm 2023-11-17T06:10:23.842088+0000 mgr.smithi006.odexej (mgr.14176) 1705 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: cephadm 2023-11-17T06:10:23.844437+0000 mgr.smithi006.odexej (mgr.14176) 1706 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.430 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.432 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.432 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.432 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: cephadm 2023-11-17T06:10:23.849972+0000 mgr.smithi006.odexej (mgr.14176) 1707 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.fizbmw on smithi006 2023-11-17T06:10:25.432 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:25 smithi006 bash[26022]: cluster 2023-11-17T06:10:24.108902+0000 mgr.smithi006.odexej (mgr.14176) 1708 : cluster [DBG] pgmap v1166: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: cephadm 2023-11-17T06:10:23.841565+0000 mgr.smithi006.odexej (mgr.14176) 1704 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:10:25.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: cephadm 2023-11-17T06:10:23.842088+0000 mgr.smithi006.odexej (mgr.14176) 1705 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: cephadm 2023-11-17T06:10:23.844437+0000 mgr.smithi006.odexej (mgr.14176) 1706 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl 2023-11-17T06:10:25.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.izkwpl ... 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: cephadm 2023-11-17T06:10:23.849972+0000 mgr.smithi006.odexej (mgr.14176) 1707 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.fizbmw on smithi006 2023-11-17T06:10:25.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:25 smithi156 bash[29655]: cluster 2023-11-17T06:10:24.108902+0000 mgr.smithi006.odexej (mgr.14176) 1708 : cluster [DBG] pgmap v1166: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:27.327 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:15.981540Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.sagqyp on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-sagqyp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.sagqyp\nDeploy daemon haproxy.nfs.foo.smithi156.sagqyp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.749795Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.msyjaj on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-msyjaj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.msyjaj\nDeploy daemon haproxy.nfs.foo.smithi006.msyjaj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:09:19.752057Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jfljqu on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:27.388 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:27 smithi006 bash[26022]: cluster 2023-11-17T06:10:26.109647+0000 mgr.smithi006.odexej (mgr.14176) 1709 : cluster [DBG] pgmap v1167: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:27.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:27 smithi156 bash[29655]: cluster 2023-11-17T06:10:26.109647+0000 mgr.smithi006.odexej (mgr.14176) 1709 : cluster [DBG] pgmap v1167: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:10:28.037 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:28.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:28 smithi006 bash[26022]: audit 2023-11-17T06:10:27.992020+0000 mon.smithi006 (mon.0) 1094 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.fizbmw"}]: dispatch 2023-11-17T06:10:28.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:28 smithi156 bash[29655]: audit 2023-11-17T06:10:27.992020+0000 mon.smithi006 (mon.0) 1094 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.fizbmw"}]: dispatch 2023-11-17T06:10:29.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:29.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: audit 2023-11-17T06:10:27.308984+0000 mgr.smithi006.odexej (mgr.14176) 1710 : audit [DBG] from='client.15414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:29.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cephadm 2023-11-17T06:10:27.990947+0000 mgr.smithi006.odexej (mgr.14176) 1711 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:10:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cephadm 2023-11-17T06:10:27.991390+0000 mgr.smithi006.odexej (mgr.14176) 1712 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cephadm 2023-11-17T06:10:27.993527+0000 mgr.smithi006.odexej (mgr.14176) 1713 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cephadm 2023-11-17T06:10:27.997825+0000 mgr.smithi006.odexej (mgr.14176) 1714 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:10:29.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cephadm 2023-11-17T06:10:28.002118+0000 mgr.smithi006.odexej (mgr.14176) 1715 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:10:29.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cluster 2023-11-17T06:10:28.003334+0000 mgr.smithi006.odexej (mgr.14176) 1716 : cluster [DBG] pgmap v1168: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:10:29.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:29 smithi006 bash[26022]: cluster 2023-11-17T06:10:28.128878+0000 mon.smithi006 (mon.0) 1095 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:10:29.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: audit 2023-11-17T06:10:27.308984+0000 mgr.smithi006.odexej (mgr.14176) 1710 : audit [DBG] from='client.15414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:29.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cephadm 2023-11-17T06:10:27.990947+0000 mgr.smithi006.odexej (mgr.14176) 1711 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:10:29.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cephadm 2023-11-17T06:10:27.991390+0000 mgr.smithi006.odexej (mgr.14176) 1712 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cephadm 2023-11-17T06:10:27.993527+0000 mgr.smithi006.odexej (mgr.14176) 1713 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.fizbmw ... 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:10:29.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:10:29.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:10:29.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:10:29.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cephadm 2023-11-17T06:10:27.997825+0000 mgr.smithi006.odexej (mgr.14176) 1714 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:10:29.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cephadm 2023-11-17T06:10:28.002118+0000 mgr.smithi006.odexej (mgr.14176) 1715 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:10:29.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cluster 2023-11-17T06:10:28.003334+0000 mgr.smithi006.odexej (mgr.14176) 1716 : cluster [DBG] pgmap v1168: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:10:29.442 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:29 smithi156 bash[29655]: cluster 2023-11-17T06:10:28.128878+0000 mon.smithi006 (mon.0) 1095 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:10:31.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:31 smithi006 bash[26022]: cluster 2023-11-17T06:10:30.004135+0000 mgr.smithi006.odexej (mgr.14176) 1717 : cluster [DBG] pgmap v1169: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:10:31.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:31 smithi156 bash[29655]: cluster 2023-11-17T06:10:30.004135+0000 mgr.smithi006.odexej (mgr.14176) 1717 : cluster [DBG] pgmap v1169: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:10:32.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:32.213 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:32 smithi006 bash[26022]: audit 2023-11-17T06:10:31.423660+0000 mon.smithi006 (mon.0) 1096 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:32 smithi006 bash[26022]: cluster 2023-11-17T06:10:32.004892+0000 mgr.smithi006.odexej (mgr.14176) 1718 : cluster [DBG] pgmap v1170: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:32 smithi006 bash[26022]: audit 2023-11-17T06:10:32.199638+0000 mgr.smithi006.odexej (mgr.14176) 1719 : audit [DBG] from='client.15418 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:32 smithi156 bash[29655]: audit 2023-11-17T06:10:31.423660+0000 mon.smithi006 (mon.0) 1096 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:10:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:32 smithi156 bash[29655]: cluster 2023-11-17T06:10:32.004892+0000 mgr.smithi006.odexej (mgr.14176) 1718 : cluster [DBG] pgmap v1170: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:32 smithi156 bash[29655]: audit 2023-11-17T06:10:32.199638+0000 mgr.smithi006.odexej (mgr.14176) 1719 : audit [DBG] from='client.15418 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:32.788 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:33.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:35.127 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:35 smithi006 bash[26022]: cluster 2023-11-17T06:10:34.005564+0000 mgr.smithi006.odexej (mgr.14176) 1720 : cluster [DBG] pgmap v1171: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T06:10:35.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:35 smithi156 bash[29655]: cluster 2023-11-17T06:10:34.005564+0000 mgr.smithi006.odexej (mgr.14176) 1720 : cluster [DBG] pgmap v1171: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T06:10:36.852 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:36.853 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:37.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:37 smithi006 bash[26022]: cluster 2023-11-17T06:10:36.006902+0000 mgr.smithi006.odexej (mgr.14176) 1721 : cluster [DBG] pgmap v1172: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:37.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:37 smithi156 bash[29655]: cluster 2023-11-17T06:10:36.006902+0000 mgr.smithi006.odexej (mgr.14176) 1721 : cluster [DBG] pgmap v1172: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:37.504 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:38.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:38 smithi006 bash[26022]: audit 2023-11-17T06:10:36.839168+0000 mgr.smithi006.odexej (mgr.14176) 1722 : audit [DBG] from='client.15422 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:38.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:38 smithi156 bash[29655]: audit 2023-11-17T06:10:36.839168+0000 mgr.smithi006.odexej (mgr.14176) 1722 : audit [DBG] from='client.15422 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:38.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:39.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:39 smithi006 bash[26022]: cluster 2023-11-17T06:10:38.007999+0000 mgr.smithi006.odexej (mgr.14176) 1723 : cluster [DBG] pgmap v1173: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:39.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:39 smithi156 bash[29655]: cluster 2023-11-17T06:10:38.007999+0000 mgr.smithi006.odexej (mgr.14176) 1723 : cluster [DBG] pgmap v1173: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:10:41.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:41 smithi006 bash[26022]: cluster 2023-11-17T06:10:40.008803+0000 mgr.smithi006.odexej (mgr.14176) 1724 : cluster [DBG] pgmap v1174: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:41.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:41 smithi156 bash[29655]: cluster 2023-11-17T06:10:40.008803+0000 mgr.smithi006.odexej (mgr.14176) 1724 : cluster [DBG] pgmap v1174: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:41.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:42.167 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:42.171 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:43.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:43.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:42 smithi156 bash[29655]: audit 2023-11-17T06:10:41.536217+0000 mgr.smithi006.odexej (mgr.14176) 1725 : audit [DBG] from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:43.182 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:42 smithi156 bash[29655]: cluster 2023-11-17T06:10:42.009720+0000 mgr.smithi006.odexej (mgr.14176) 1726 : cluster [DBG] pgmap v1175: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:43.186 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:42 smithi006 bash[26022]: audit 2023-11-17T06:10:41.536217+0000 mgr.smithi006.odexej (mgr.14176) 1725 : audit [DBG] from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:43.187 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:42 smithi006 bash[26022]: cluster 2023-11-17T06:10:42.009720+0000 mgr.smithi006.odexej (mgr.14176) 1726 : cluster [DBG] pgmap v1175: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:45.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:45 smithi006 bash[26022]: cluster 2023-11-17T06:10:44.010917+0000 mgr.smithi006.odexej (mgr.14176) 1727 : cluster [DBG] pgmap v1176: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:45.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:45 smithi156 bash[29655]: cluster 2023-11-17T06:10:44.010917+0000 mgr.smithi006.odexej (mgr.14176) 1727 : cluster [DBG] pgmap v1176: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:46.505 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:46.505 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:47.087 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:47.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:47 smithi006 bash[26022]: cluster 2023-11-17T06:10:46.012330+0000 mgr.smithi006.odexej (mgr.14176) 1728 : cluster [DBG] pgmap v1177: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:47.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:47 smithi156 bash[29655]: cluster 2023-11-17T06:10:46.012330+0000 mgr.smithi006.odexej (mgr.14176) 1728 : cluster [DBG] pgmap v1177: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:48.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:48.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:48 smithi006 bash[26022]: audit 2023-11-17T06:10:46.488419+0000 mgr.smithi006.odexej (mgr.14176) 1729 : audit [DBG] from='client.15430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:48.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:48 smithi156 bash[29655]: audit 2023-11-17T06:10:46.488419+0000 mgr.smithi006.odexej (mgr.14176) 1729 : audit [DBG] from='client.15430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:49.389 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:49 smithi006 bash[26022]: cluster 2023-11-17T06:10:48.013387+0000 mgr.smithi006.odexej (mgr.14176) 1730 : cluster [DBG] pgmap v1178: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:49.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:49 smithi156 bash[29655]: cluster 2023-11-17T06:10:48.013387+0000 mgr.smithi006.odexej (mgr.14176) 1730 : cluster [DBG] pgmap v1178: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:51.103 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:51.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:51 smithi006 bash[26022]: cluster 2023-11-17T06:10:50.014810+0000 mgr.smithi006.odexej (mgr.14176) 1731 : cluster [DBG] pgmap v1179: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:51.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:51 smithi156 bash[29655]: cluster 2023-11-17T06:10:50.014810+0000 mgr.smithi006.odexej (mgr.14176) 1731 : cluster [DBG] pgmap v1179: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:51.711 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:52.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:52 smithi006 bash[26022]: audit 2023-11-17T06:10:51.079164+0000 mgr.smithi006.odexej (mgr.14176) 1732 : audit [DBG] from='client.15434 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:52.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:52 smithi156 bash[29655]: audit 2023-11-17T06:10:51.079164+0000 mgr.smithi006.odexej (mgr.14176) 1732 : audit [DBG] from='client.15434 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:52.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:53.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:53 smithi006 bash[26022]: cluster 2023-11-17T06:10:52.015845+0000 mgr.smithi006.odexej (mgr.14176) 1733 : cluster [DBG] pgmap v1180: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:53.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:53 smithi156 bash[29655]: cluster 2023-11-17T06:10:52.015845+0000 mgr.smithi006.odexej (mgr.14176) 1733 : cluster [DBG] pgmap v1180: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:55.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:55 smithi006 bash[26022]: cluster 2023-11-17T06:10:54.017037+0000 mgr.smithi006.odexej (mgr.14176) 1734 : cluster [DBG] pgmap v1181: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:55.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:55 smithi156 bash[29655]: cluster 2023-11-17T06:10:54.017037+0000 mgr.smithi006.odexej (mgr.14176) 1734 : cluster [DBG] pgmap v1181: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:10:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:10:55.799 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:10:56.410 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:10:57.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:10:57.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:57 smithi006 bash[26022]: audit 2023-11-17T06:10:55.782765+0000 mgr.smithi006.odexej (mgr.14176) 1735 : audit [DBG] from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:57.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:57 smithi006 bash[26022]: cluster 2023-11-17T06:10:56.018428+0000 mgr.smithi006.odexej (mgr.14176) 1736 : cluster [DBG] pgmap v1182: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:57.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:57 smithi156 bash[29655]: audit 2023-11-17T06:10:55.782765+0000 mgr.smithi006.odexej (mgr.14176) 1735 : audit [DBG] from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:10:57.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:57 smithi156 bash[29655]: cluster 2023-11-17T06:10:56.018428+0000 mgr.smithi006.odexej (mgr.14176) 1736 : cluster [DBG] pgmap v1182: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:59.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:10:59 smithi006 bash[26022]: cluster 2023-11-17T06:10:58.019479+0000 mgr.smithi006.odexej (mgr.14176) 1737 : cluster [DBG] pgmap v1183: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:10:59.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:10:59 smithi156 bash[29655]: cluster 2023-11-17T06:10:58.019479+0000 mgr.smithi006.odexej (mgr.14176) 1737 : cluster [DBG] pgmap v1183: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:00.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:02.448 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:02.449 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:01 smithi156 bash[29655]: cluster 2023-11-17T06:11:00.020692+0000 mgr.smithi006.odexej (mgr.14176) 1738 : cluster [DBG] pgmap v1184: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:02.450 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:02 smithi156 bash[29655]: audit 2023-11-17T06:11:00.471237+0000 mgr.smithi006.odexej (mgr.14176) 1739 : audit [DBG] from='client.15442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:02.450 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:02.451 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:01 smithi006 bash[26022]: cluster 2023-11-17T06:11:00.020692+0000 mgr.smithi006.odexej (mgr.14176) 1738 : cluster [DBG] pgmap v1184: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:02.451 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:02 smithi006 bash[26022]: audit 2023-11-17T06:11:00.471237+0000 mgr.smithi006.odexej (mgr.14176) 1739 : audit [DBG] from='client.15442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:03.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:03 smithi006 bash[26022]: cluster 2023-11-17T06:11:02.021825+0000 mgr.smithi006.odexej (mgr.14176) 1740 : cluster [DBG] pgmap v1185: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:03.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:03 smithi156 bash[29655]: cluster 2023-11-17T06:11:02.021825+0000 mgr.smithi006.odexej (mgr.14176) 1740 : cluster [DBG] pgmap v1185: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:03.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:05.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:05 smithi006 bash[26022]: cluster 2023-11-17T06:11:04.023018+0000 mgr.smithi006.odexej (mgr.14176) 1741 : cluster [DBG] pgmap v1186: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:05.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:05 smithi156 bash[29655]: cluster 2023-11-17T06:11:04.023018+0000 mgr.smithi006.odexej (mgr.14176) 1741 : cluster [DBG] pgmap v1186: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:06.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:06.631 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:07.241 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:07.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:07 smithi006 bash[26022]: cluster 2023-11-17T06:11:06.024591+0000 mgr.smithi006.odexej (mgr.14176) 1742 : cluster [DBG] pgmap v1187: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:07.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:07 smithi156 bash[29655]: cluster 2023-11-17T06:11:06.024591+0000 mgr.smithi006.odexej (mgr.14176) 1742 : cluster [DBG] pgmap v1187: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:08.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:08.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:08 smithi006 bash[26022]: audit 2023-11-17T06:11:06.617523+0000 mgr.smithi006.odexej (mgr.14176) 1743 : audit [DBG] from='client.15446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:08.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:08 smithi156 bash[29655]: audit 2023-11-17T06:11:06.617523+0000 mgr.smithi006.odexej (mgr.14176) 1743 : audit [DBG] from='client.15446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:09.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:09 smithi006 bash[26022]: cluster 2023-11-17T06:11:08.025842+0000 mgr.smithi006.odexej (mgr.14176) 1744 : cluster [DBG] pgmap v1188: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:09.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:09 smithi156 bash[29655]: cluster 2023-11-17T06:11:08.025842+0000 mgr.smithi006.odexej (mgr.14176) 1744 : cluster [DBG] pgmap v1188: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:11.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:11.239 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:11.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:11 smithi156 bash[29655]: cluster 2023-11-17T06:11:10.027296+0000 mgr.smithi006.odexej (mgr.14176) 1745 : cluster [DBG] pgmap v1189: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:11.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:11 smithi006 bash[26022]: cluster 2023-11-17T06:11:10.027296+0000 mgr.smithi006.odexej (mgr.14176) 1745 : cluster [DBG] pgmap v1189: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:11.894 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:12.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:12 smithi156 bash[29655]: audit 2023-11-17T06:11:11.227863+0000 mgr.smithi006.odexej (mgr.14176) 1746 : audit [DBG] from='client.15450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:12.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:12 smithi006 bash[26022]: audit 2023-11-17T06:11:11.227863+0000 mgr.smithi006.odexej (mgr.14176) 1746 : audit [DBG] from='client.15450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:12.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:13.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:13 smithi156 bash[29655]: cluster 2023-11-17T06:11:12.028361+0000 mgr.smithi006.odexej (mgr.14176) 1747 : cluster [DBG] pgmap v1190: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:13.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:13 smithi006 bash[26022]: cluster 2023-11-17T06:11:12.028361+0000 mgr.smithi006.odexej (mgr.14176) 1747 : cluster [DBG] pgmap v1190: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:15.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:15 smithi006 bash[26022]: cluster 2023-11-17T06:11:14.029541+0000 mgr.smithi006.odexej (mgr.14176) 1748 : cluster [DBG] pgmap v1191: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:15.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:15 smithi156 bash[29655]: cluster 2023-11-17T06:11:14.029541+0000 mgr.smithi006.odexej (mgr.14176) 1748 : cluster [DBG] pgmap v1191: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:16.008 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:16.620 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:17.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:17.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:17 smithi006 bash[26022]: audit 2023-11-17T06:11:15.995647+0000 mgr.smithi006.odexej (mgr.14176) 1749 : audit [DBG] from='client.15454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:17.632 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:17 smithi006 bash[26022]: cluster 2023-11-17T06:11:16.030830+0000 mgr.smithi006.odexej (mgr.14176) 1750 : cluster [DBG] pgmap v1192: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:17.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:17 smithi156 bash[29655]: audit 2023-11-17T06:11:15.995647+0000 mgr.smithi006.odexej (mgr.14176) 1749 : audit [DBG] from='client.15454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:17.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:17 smithi156 bash[29655]: cluster 2023-11-17T06:11:16.030830+0000 mgr.smithi006.odexej (mgr.14176) 1750 : cluster [DBG] pgmap v1192: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:19.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:19 smithi006 bash[26022]: cluster 2023-11-17T06:11:18.031905+0000 mgr.smithi006.odexej (mgr.14176) 1751 : cluster [DBG] pgmap v1193: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:19.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:19 smithi156 bash[29655]: cluster 2023-11-17T06:11:18.031905+0000 mgr.smithi006.odexej (mgr.14176) 1751 : cluster [DBG] pgmap v1193: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:20.755 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:20.757 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:21.364 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:21.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:21 smithi006 bash[26022]: cluster 2023-11-17T06:11:20.033393+0000 mgr.smithi006.odexej (mgr.14176) 1752 : cluster [DBG] pgmap v1194: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:21 smithi156 bash[29655]: cluster 2023-11-17T06:11:20.033393+0000 mgr.smithi006.odexej (mgr.14176) 1752 : cluster [DBG] pgmap v1194: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:22.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:22.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:22 smithi006 bash[26022]: audit 2023-11-17T06:11:20.737586+0000 mgr.smithi006.odexej (mgr.14176) 1753 : audit [DBG] from='client.15458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:22 smithi156 bash[29655]: audit 2023-11-17T06:11:20.737586+0000 mgr.smithi006.odexej (mgr.14176) 1753 : audit [DBG] from='client.15458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:23.474 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:23 smithi006 bash[26022]: cluster 2023-11-17T06:11:22.034507+0000 mgr.smithi006.odexej (mgr.14176) 1754 : cluster [DBG] pgmap v1195: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:23.624 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:23 smithi156 bash[29655]: cluster 2023-11-17T06:11:22.034507+0000 mgr.smithi006.odexej (mgr.14176) 1754 : cluster [DBG] pgmap v1195: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:25.324 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:25.324 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:25.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:25 smithi006 bash[26022]: cluster 2023-11-17T06:11:24.035707+0000 mgr.smithi006.odexej (mgr.14176) 1755 : cluster [DBG] pgmap v1196: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:25.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:25 smithi156 bash[29655]: cluster 2023-11-17T06:11:24.035707+0000 mgr.smithi006.odexej (mgr.14176) 1755 : cluster [DBG] pgmap v1196: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:11:25.916 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:26.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:26 smithi006 bash[26022]: audit 2023-11-17T06:11:25.314811+0000 mgr.smithi006.odexej (mgr.14176) 1756 : audit [DBG] from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:26.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:26 smithi156 bash[29655]: audit 2023-11-17T06:11:25.314811+0000 mgr.smithi006.odexej (mgr.14176) 1756 : audit [DBG] from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:26.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:27.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:27 smithi006 bash[26022]: cluster 2023-11-17T06:11:26.037100+0000 mgr.smithi006.odexej (mgr.14176) 1757 : cluster [DBG] pgmap v1197: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:27.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:27 smithi156 bash[29655]: cluster 2023-11-17T06:11:26.037100+0000 mgr.smithi006.odexej (mgr.14176) 1757 : cluster [DBG] pgmap v1197: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:28.309 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:28 smithi006 bash[26022]: audit 2023-11-17T06:11:28.006258+0000 mon.smithi006 (mon.0) 1097 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:11:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:28 smithi156 bash[29655]: audit 2023-11-17T06:11:28.006258+0000 mon.smithi006 (mon.0) 1097 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:11:29.620 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:29 smithi006 bash[26022]: cluster 2023-11-17T06:11:28.037895+0000 mgr.smithi006.odexej (mgr.14176) 1758 : cluster [DBG] pgmap v1198: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:29.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:29 smithi156 bash[29655]: cluster 2023-11-17T06:11:28.037895+0000 mgr.smithi006.odexej (mgr.14176) 1758 : cluster [DBG] pgmap v1198: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:30.058 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:11:30.058 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:11:30.662 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:11:31.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:11:31.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:31 smithi006 bash[26022]: cluster 2023-11-17T06:11:30.039362+0000 mgr.smithi006.odexej (mgr.14176) 1759 : cluster [DBG] pgmap v1199: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:31.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:31 smithi006 bash[26022]: audit 2023-11-17T06:11:30.041794+0000 mgr.smithi006.odexej (mgr.14176) 1760 : audit [DBG] from='client.15466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:31.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:31 smithi156 bash[29655]: cluster 2023-11-17T06:11:30.039362+0000 mgr.smithi006.odexej (mgr.14176) 1759 : cluster [DBG] pgmap v1199: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:31.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:31 smithi156 bash[29655]: audit 2023-11-17T06:11:30.041794+0000 mgr.smithi006.odexej (mgr.14176) 1760 : audit [DBG] from='client.15466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:11:33.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:32 smithi006 bash[26022]: cluster 2023-11-17T06:11:32.040501+0000 mgr.smithi006.odexej (mgr.14176) 1761 : cluster [DBG] pgmap v1200: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:33.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:32 smithi156 bash[29655]: cluster 2023-11-17T06:11:32.040501+0000 mgr.smithi006.odexej (mgr.14176) 1761 : cluster [DBG] pgmap v1200: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:11:35.051 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:13.543 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:09:19.753701Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.djgbjb on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:23.844201Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.izkwpl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-izkwpl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.izkwpl\nDeploy daemon haproxy.nfs.foo.smithi156.izkwpl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.993262Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.fizbmw on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-fizbmw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.fizbmw\nDeploy daemon haproxy.nfs.foo.smithi006.fizbmw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:10:27.997595Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bqlizp on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:13.545 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:35 smithi006 bash[26022]: cluster 2023-11-17T06:11:34.041674+0000 mgr.smithi006.odexej (mgr.14176) 1762 : cluster [DBG] pgmap v1201: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:13.554 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:35 smithi156 bash[29655]: cluster 2023-11-17T06:11:34.041674+0000 mgr.smithi006.odexej (mgr.14176) 1762 : cluster [DBG] pgmap v1201: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:36 smithi156 bash[29655]: audit 2023-11-17T06:11:35.041934+0000 mgr.smithi006.odexej (mgr.14176) 1763 : audit [DBG] from='client.15470 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:37 smithi156 bash[29655]: cluster 2023-11-17T06:11:36.043087+0000 mgr.smithi006.odexej (mgr.14176) 1764 : cluster [DBG] pgmap v1202: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:38 smithi156 bash[29655]: audit 2023-11-17T06:11:37.756674+0000 mon.smithi006 (mon.0) 1098 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:38 smithi156 bash[29655]: cluster 2023-11-17T06:11:38.044149+0000 mgr.smithi006.odexej (mgr.14176) 1765 : cluster [DBG] pgmap v1203: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:38.850780+0000 mon.smithi006 (mon.0) 1099 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:39.138550+0000 mon.smithi006 (mon.0) 1100 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:12:13.558 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:39.139886+0000 mon.smithi006 (mon.0) 1101 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:12:13.598 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:39.148514+0000 mon.smithi006 (mon.0) 1102 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:13.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: cluster 2023-11-17T06:11:39.149419+0000 mgr.smithi006.odexej (mgr.14176) 1766 : cluster [DBG] pgmap v1204: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:13.626 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:39.156473+0000 mon.smithi006 (mon.0) 1103 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:12:13.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: audit 2023-11-17T06:11:39.171192+0000 mon.smithi006 (mon.0) 1104 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:13.842 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:39 smithi156 bash[29655]: cephadm 2023-11-17T06:11:39.174083+0000 mgr.smithi006.odexej (mgr.14176) 1767 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.cjxqzl on smithi156 2023-11-17T06:12:13.842 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:40 smithi156 bash[29655]: cluster 2023-11-17T06:11:40.144745+0000 mon.smithi006 (mon.0) 1105 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:12:13.842 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:40 smithi156 bash[29655]: cluster 2023-11-17T06:11:40.144814+0000 mon.smithi006 (mon.0) 1106 : cluster [INF] Cluster is now healthy 2023-11-17T06:12:13.842 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:41 smithi156 bash[29655]: cluster 2023-11-17T06:11:41.150512+0000 mgr.smithi006.odexej (mgr.14176) 1768 : cluster [DBG] pgmap v1205: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:13.843 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:42 smithi156 bash[29655]: audit 2023-11-17T06:11:42.901296+0000 mon.smithi006 (mon.0) 1107 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.cjxqzl"}]: dispatch 2023-11-17T06:12:13.843 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: cephadm 2023-11-17T06:11:42.899948+0000 mgr.smithi006.odexej (mgr.14176) 1769 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:13.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:13.887 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: cephadm 2023-11-17T06:11:42.900578+0000 mgr.smithi006.odexej (mgr.14176) 1770 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: cephadm 2023-11-17T06:11:42.903091+0000 mgr.smithi006.odexej (mgr.14176) 1771 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.888 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:13.889 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: cephadm 2023-11-17T06:11:42.908006+0000 mgr.smithi006.odexej (mgr.14176) 1772 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.aneiee on smithi006 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:43 smithi156 bash[29655]: cluster 2023-11-17T06:11:43.151277+0000 mgr.smithi006.odexej (mgr.14176) 1773 : cluster [DBG] pgmap v1206: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 184 B/s rd, 0 op/s 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:46 smithi156 bash[29655]: cluster 2023-11-17T06:11:45.152432+0000 mgr.smithi006.odexej (mgr.14176) 1774 : cluster [DBG] pgmap v1207: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:13.913 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:47 smithi156 bash[29655]: audit 2023-11-17T06:11:46.640445+0000 mon.smithi006 (mon.0) 1108 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.aneiee"}]: dispatch 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cephadm 2023-11-17T06:11:46.639067+0000 mgr.smithi006.odexej (mgr.14176) 1775 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:13.914 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:13.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:13.916 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:13.916 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:13.916 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.040 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.041 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.042 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cephadm 2023-11-17T06:11:46.639697+0000 mgr.smithi006.odexej (mgr.14176) 1776 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cephadm 2023-11-17T06:11:46.641982+0000 mgr.smithi006.odexej (mgr.14176) 1777 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.084 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cephadm 2023-11-17T06:11:46.646241+0000 mgr.smithi006.odexej (mgr.14176) 1778 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cephadm 2023-11-17T06:11:46.650426+0000 mgr.smithi006.odexej (mgr.14176) 1779 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cluster 2023-11-17T06:11:46.651683+0000 mgr.smithi006.odexej (mgr.14176) 1780 : cluster [DBG] pgmap v1208: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:48 smithi156 bash[29655]: cluster 2023-11-17T06:11:47.210661+0000 mon.smithi006 (mon.0) 1109 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:50 smithi156 bash[29655]: cluster 2023-11-17T06:11:48.652876+0000 mgr.smithi006.odexej (mgr.14176) 1781 : cluster [DBG] pgmap v1209: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:52 smithi156 bash[29655]: cluster 2023-11-17T06:11:50.654259+0000 mgr.smithi006.odexej (mgr.14176) 1782 : cluster [DBG] pgmap v1210: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:52 smithi156 bash[29655]: audit 2023-11-17T06:11:51.487278+0000 mon.smithi006 (mon.0) 1110 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:54 smithi156 bash[29655]: cluster 2023-11-17T06:11:52.655306+0000 mgr.smithi006.odexej (mgr.14176) 1783 : cluster [DBG] pgmap v1211: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:56 smithi156 bash[29655]: cluster 2023-11-17T06:11:54.656708+0000 mgr.smithi006.odexej (mgr.14176) 1784 : cluster [DBG] pgmap v1212: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:11:58 smithi156 bash[29655]: cluster 2023-11-17T06:11:56.657699+0000 mgr.smithi006.odexej (mgr.14176) 1785 : cluster [DBG] pgmap v1213: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:00 smithi156 bash[29655]: cluster 2023-11-17T06:11:58.658900+0000 mgr.smithi006.odexej (mgr.14176) 1786 : cluster [DBG] pgmap v1214: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:02 smithi156 bash[29655]: cluster 2023-11-17T06:12:00.660286+0000 mgr.smithi006.odexej (mgr.14176) 1787 : cluster [DBG] pgmap v1215: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:04 smithi156 bash[29655]: cluster 2023-11-17T06:12:02.661246+0000 mgr.smithi006.odexej (mgr.14176) 1788 : cluster [DBG] pgmap v1216: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:14.085 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:06 smithi156 bash[29655]: cluster 2023-11-17T06:12:04.662630+0000 mgr.smithi006.odexej (mgr.14176) 1789 : cluster [DBG] pgmap v1217: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.086 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:08 smithi156 bash[29655]: cluster 2023-11-17T06:12:06.663675+0000 mgr.smithi006.odexej (mgr.14176) 1790 : cluster [DBG] pgmap v1218: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.086 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:10 smithi156 bash[29655]: cluster 2023-11-17T06:12:08.664839+0000 mgr.smithi006.odexej (mgr.14176) 1791 : cluster [DBG] pgmap v1219: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.086 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:12 smithi156 bash[29655]: cluster 2023-11-17T06:12:10.666189+0000 mgr.smithi006.odexej (mgr.14176) 1792 : cluster [DBG] pgmap v1220: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:36 smithi006 bash[26022]: audit 2023-11-17T06:11:35.041934+0000 mgr.smithi006.odexej (mgr.14176) 1763 : audit [DBG] from='client.15470 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:37 smithi006 bash[26022]: cluster 2023-11-17T06:11:36.043087+0000 mgr.smithi006.odexej (mgr.14176) 1764 : cluster [DBG] pgmap v1202: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:38 smithi006 bash[26022]: audit 2023-11-17T06:11:37.756674+0000 mon.smithi006 (mon.0) 1098 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:38 smithi006 bash[26022]: cluster 2023-11-17T06:11:38.044149+0000 mgr.smithi006.odexej (mgr.14176) 1765 : cluster [DBG] pgmap v1203: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:38.850780+0000 mon.smithi006 (mon.0) 1099 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:39.138550+0000 mon.smithi006 (mon.0) 1100 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:39.139886+0000 mon.smithi006 (mon.0) 1101 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:39.148514+0000 mon.smithi006 (mon.0) 1102 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.087 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: cluster 2023-11-17T06:11:39.149419+0000 mgr.smithi006.odexej (mgr.14176) 1766 : cluster [DBG] pgmap v1204: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:39.156473+0000 mon.smithi006 (mon.0) 1103 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: audit 2023-11-17T06:11:39.171192+0000 mon.smithi006 (mon.0) 1104 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:39 smithi006 bash[26022]: cephadm 2023-11-17T06:11:39.174083+0000 mgr.smithi006.odexej (mgr.14176) 1767 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.cjxqzl on smithi156 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:40 smithi006 bash[26022]: cluster 2023-11-17T06:11:40.144745+0000 mon.smithi006 (mon.0) 1105 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:40 smithi006 bash[26022]: cluster 2023-11-17T06:11:40.144814+0000 mon.smithi006 (mon.0) 1106 : cluster [INF] Cluster is now healthy 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:41 smithi006 bash[26022]: cluster 2023-11-17T06:11:41.150512+0000 mgr.smithi006.odexej (mgr.14176) 1768 : cluster [DBG] pgmap v1205: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:42 smithi006 bash[26022]: audit 2023-11-17T06:11:42.901296+0000 mon.smithi006 (mon.0) 1107 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.cjxqzl"}]: dispatch 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: cephadm 2023-11-17T06:11:42.899948+0000 mgr.smithi006.odexej (mgr.14176) 1769 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.088 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.089 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: cephadm 2023-11-17T06:11:42.900578+0000 mgr.smithi006.odexej (mgr.14176) 1770 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: cephadm 2023-11-17T06:11:42.903091+0000 mgr.smithi006.odexej (mgr.14176) 1771 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl 2023-11-17T06:12:14.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.cjxqzl ... 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: cephadm 2023-11-17T06:11:42.908006+0000 mgr.smithi006.odexej (mgr.14176) 1772 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.aneiee on smithi006 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:43 smithi006 bash[26022]: cluster 2023-11-17T06:11:43.151277+0000 mgr.smithi006.odexej (mgr.14176) 1773 : cluster [DBG] pgmap v1206: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 184 B/s rd, 0 op/s 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:46 smithi006 bash[26022]: cluster 2023-11-17T06:11:45.152432+0000 mgr.smithi006.odexej (mgr.14176) 1774 : cluster [DBG] pgmap v1207: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:47 smithi006 bash[26022]: audit 2023-11-17T06:11:46.640445+0000 mon.smithi006 (mon.0) 1108 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.aneiee"}]: dispatch 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cephadm 2023-11-17T06:11:46.639067+0000 mgr.smithi006.odexej (mgr.14176) 1775 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:12:14.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cephadm 2023-11-17T06:11:46.639697+0000 mgr.smithi006.odexej (mgr.14176) 1776 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cephadm 2023-11-17T06:11:46.641982+0000 mgr.smithi006.odexej (mgr.14176) 1777 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.aneiee ... 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:14.094 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cephadm 2023-11-17T06:11:46.646241+0000 mgr.smithi006.odexej (mgr.14176) 1778 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cephadm 2023-11-17T06:11:46.650426+0000 mgr.smithi006.odexej (mgr.14176) 1779 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cluster 2023-11-17T06:11:46.651683+0000 mgr.smithi006.odexej (mgr.14176) 1780 : cluster [DBG] pgmap v1208: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:48 smithi006 bash[26022]: cluster 2023-11-17T06:11:47.210661+0000 mon.smithi006 (mon.0) 1109 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:50 smithi006 bash[26022]: cluster 2023-11-17T06:11:48.652876+0000 mgr.smithi006.odexej (mgr.14176) 1781 : cluster [DBG] pgmap v1209: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:52 smithi006 bash[26022]: cluster 2023-11-17T06:11:50.654259+0000 mgr.smithi006.odexej (mgr.14176) 1782 : cluster [DBG] pgmap v1210: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:52 smithi006 bash[26022]: audit 2023-11-17T06:11:51.487278+0000 mon.smithi006 (mon.0) 1110 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:54 smithi006 bash[26022]: cluster 2023-11-17T06:11:52.655306+0000 mgr.smithi006.odexej (mgr.14176) 1783 : cluster [DBG] pgmap v1211: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:56 smithi006 bash[26022]: cluster 2023-11-17T06:11:54.656708+0000 mgr.smithi006.odexej (mgr.14176) 1784 : cluster [DBG] pgmap v1212: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:11:58 smithi006 bash[26022]: cluster 2023-11-17T06:11:56.657699+0000 mgr.smithi006.odexej (mgr.14176) 1785 : cluster [DBG] pgmap v1213: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:00 smithi006 bash[26022]: cluster 2023-11-17T06:11:58.658900+0000 mgr.smithi006.odexej (mgr.14176) 1786 : cluster [DBG] pgmap v1214: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:02 smithi006 bash[26022]: cluster 2023-11-17T06:12:00.660286+0000 mgr.smithi006.odexej (mgr.14176) 1787 : cluster [DBG] pgmap v1215: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:04 smithi006 bash[26022]: cluster 2023-11-17T06:12:02.661246+0000 mgr.smithi006.odexej (mgr.14176) 1788 : cluster [DBG] pgmap v1216: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:06 smithi006 bash[26022]: cluster 2023-11-17T06:12:04.662630+0000 mgr.smithi006.odexej (mgr.14176) 1789 : cluster [DBG] pgmap v1217: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.095 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:08 smithi006 bash[26022]: cluster 2023-11-17T06:12:06.663675+0000 mgr.smithi006.odexej (mgr.14176) 1790 : cluster [DBG] pgmap v1218: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:10 smithi006 bash[26022]: cluster 2023-11-17T06:12:08.664839+0000 mgr.smithi006.odexej (mgr.14176) 1791 : cluster [DBG] pgmap v1219: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.096 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:12 smithi006 bash[26022]: cluster 2023-11-17T06:12:10.666189+0000 mgr.smithi006.odexej (mgr.14176) 1792 : cluster [DBG] pgmap v1220: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:14.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:14.599 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:14 smithi006 bash[26022]: cluster 2023-11-17T06:12:12.667251+0000 mgr.smithi006.odexej (mgr.14176) 1793 : cluster [DBG] pgmap v1221: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:14.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:14 smithi156 bash[29655]: cluster 2023-11-17T06:12:12.667251+0000 mgr.smithi006.odexej (mgr.14176) 1793 : cluster [DBG] pgmap v1221: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:16.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:16 smithi006 bash[26022]: cluster 2023-11-17T06:12:14.668696+0000 mgr.smithi006.odexej (mgr.14176) 1794 : cluster [DBG] pgmap v1222: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:16.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:16 smithi156 bash[29655]: cluster 2023-11-17T06:12:14.668696+0000 mgr.smithi006.odexej (mgr.14176) 1794 : cluster [DBG] pgmap v1222: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:17.725 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:17.726 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:18.325 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:18.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:18 smithi006 bash[26022]: cluster 2023-11-17T06:12:16.669724+0000 mgr.smithi006.odexej (mgr.14176) 1795 : cluster [DBG] pgmap v1223: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:18.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:18 smithi156 bash[29655]: cluster 2023-11-17T06:12:16.669724+0000 mgr.smithi006.odexej (mgr.14176) 1795 : cluster [DBG] pgmap v1223: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:19.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:19.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:19 smithi006 bash[26022]: audit 2023-11-17T06:12:17.708827+0000 mgr.smithi006.odexej (mgr.14176) 1796 : audit [DBG] from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:19.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:19 smithi156 bash[29655]: audit 2023-11-17T06:12:17.708827+0000 mgr.smithi006.odexej (mgr.14176) 1796 : audit [DBG] from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:20.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:20 smithi006 bash[26022]: cluster 2023-11-17T06:12:18.670907+0000 mgr.smithi006.odexej (mgr.14176) 1797 : cluster [DBG] pgmap v1224: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:20 smithi156 bash[29655]: cluster 2023-11-17T06:12:18.670907+0000 mgr.smithi006.odexej (mgr.14176) 1797 : cluster [DBG] pgmap v1224: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:21.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:21 smithi006 bash[26022]: cluster 2023-11-17T06:12:20.672697+0000 mgr.smithi006.odexej (mgr.14176) 1798 : cluster [DBG] pgmap v1225: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:21 smithi156 bash[29655]: cluster 2023-11-17T06:12:20.672697+0000 mgr.smithi006.odexej (mgr.14176) 1798 : cluster [DBG] pgmap v1225: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:22.472 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:23.080 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:24.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:24.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:23 smithi006 bash[26022]: audit 2023-11-17T06:12:22.456165+0000 mgr.smithi006.odexej (mgr.14176) 1799 : audit [DBG] from='client.15478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:24.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:23 smithi006 bash[26022]: cluster 2023-11-17T06:12:22.673841+0000 mgr.smithi006.odexej (mgr.14176) 1800 : cluster [DBG] pgmap v1226: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:24.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:23 smithi156 bash[29655]: audit 2023-11-17T06:12:22.456165+0000 mgr.smithi006.odexej (mgr.14176) 1799 : audit [DBG] from='client.15478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:24.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:23 smithi156 bash[29655]: cluster 2023-11-17T06:12:22.673841+0000 mgr.smithi006.odexej (mgr.14176) 1800 : cluster [DBG] pgmap v1226: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:26.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:25 smithi006 bash[26022]: cluster 2023-11-17T06:12:24.675439+0000 mgr.smithi006.odexej (mgr.14176) 1801 : cluster [DBG] pgmap v1227: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:26.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:25 smithi156 bash[29655]: cluster 2023-11-17T06:12:24.675439+0000 mgr.smithi006.odexej (mgr.14176) 1801 : cluster [DBG] pgmap v1227: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:27.225 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:27.225 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:27.822 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:28.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:27 smithi006 bash[26022]: cluster 2023-11-17T06:12:26.676610+0000 mgr.smithi006.odexej (mgr.14176) 1802 : cluster [DBG] pgmap v1228: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:28.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:27 smithi006 bash[26022]: audit 2023-11-17T06:12:27.209106+0000 mgr.smithi006.odexej (mgr.14176) 1803 : audit [DBG] from='client.15482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:28.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:27 smithi156 bash[29655]: cluster 2023-11-17T06:12:26.676610+0000 mgr.smithi006.odexej (mgr.14176) 1802 : cluster [DBG] pgmap v1228: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:28.432 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:27 smithi156 bash[29655]: audit 2023-11-17T06:12:27.209106+0000 mgr.smithi006.odexej (mgr.14176) 1803 : audit [DBG] from='client.15482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:28.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:30.110 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:29 smithi006 bash[26022]: cluster 2023-11-17T06:12:28.677788+0000 mgr.smithi006.odexej (mgr.14176) 1804 : cluster [DBG] pgmap v1229: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:30.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:29 smithi156 bash[29655]: cluster 2023-11-17T06:12:28.677788+0000 mgr.smithi006.odexej (mgr.14176) 1804 : cluster [DBG] pgmap v1229: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:31.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:31.773 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:32.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:31 smithi006 bash[26022]: cluster 2023-11-17T06:12:30.679188+0000 mgr.smithi006.odexej (mgr.14176) 1805 : cluster [DBG] pgmap v1230: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:32.402 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:32.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:31 smithi156 bash[29655]: cluster 2023-11-17T06:12:30.679188+0000 mgr.smithi006.odexej (mgr.14176) 1805 : cluster [DBG] pgmap v1230: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:33.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:33.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:32 smithi006 bash[26022]: audit 2023-11-17T06:12:31.758280+0000 mgr.smithi006.odexej (mgr.14176) 1806 : audit [DBG] from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:33.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:32 smithi156 bash[29655]: audit 2023-11-17T06:12:31.758280+0000 mgr.smithi006.odexej (mgr.14176) 1806 : audit [DBG] from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:34.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:33 smithi006 bash[26022]: cluster 2023-11-17T06:12:32.680440+0000 mgr.smithi006.odexej (mgr.14176) 1807 : cluster [DBG] pgmap v1231: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:34.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:33 smithi156 bash[29655]: cluster 2023-11-17T06:12:32.680440+0000 mgr.smithi006.odexej (mgr.14176) 1807 : cluster [DBG] pgmap v1231: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:36.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:35 smithi006 bash[26022]: cluster 2023-11-17T06:12:34.681858+0000 mgr.smithi006.odexej (mgr.14176) 1808 : cluster [DBG] pgmap v1232: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:36.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:36 smithi156 bash[29655]: cluster 2023-11-17T06:12:34.681858+0000 mgr.smithi006.odexej (mgr.14176) 1808 : cluster [DBG] pgmap v1232: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:36.500 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:37.148 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:38.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:38.275 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:38 smithi006 bash[26022]: audit 2023-11-17T06:12:36.488907+0000 mgr.smithi006.odexej (mgr.14176) 1809 : audit [DBG] from='client.15490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:38.275 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:38 smithi006 bash[26022]: cluster 2023-11-17T06:12:36.682922+0000 mgr.smithi006.odexej (mgr.14176) 1810 : cluster [DBG] pgmap v1233: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:38.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:38 smithi156 bash[29655]: audit 2023-11-17T06:12:36.488907+0000 mgr.smithi006.odexej (mgr.14176) 1809 : audit [DBG] from='client.15490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:38.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:38 smithi156 bash[29655]: cluster 2023-11-17T06:12:36.682922+0000 mgr.smithi006.odexej (mgr.14176) 1810 : cluster [DBG] pgmap v1233: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:40.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:40 smithi006 bash[26022]: cluster 2023-11-17T06:12:38.684209+0000 mgr.smithi006.odexej (mgr.14176) 1811 : cluster [DBG] pgmap v1234: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:40.495 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:40 smithi156 bash[29655]: cluster 2023-11-17T06:12:38.684209+0000 mgr.smithi006.odexej (mgr.14176) 1811 : cluster [DBG] pgmap v1234: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:41.379 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:42.029 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:42.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:42 smithi006 bash[26022]: cluster 2023-11-17T06:12:40.685655+0000 mgr.smithi006.odexej (mgr.14176) 1812 : cluster [DBG] pgmap v1235: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:42.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:42 smithi156 bash[29655]: cluster 2023-11-17T06:12:40.685655+0000 mgr.smithi006.odexej (mgr.14176) 1812 : cluster [DBG] pgmap v1235: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:43.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:43.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:43 smithi006 bash[26022]: audit 2023-11-17T06:12:41.360961+0000 mgr.smithi006.odexej (mgr.14176) 1813 : audit [DBG] from='client.15494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:43.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:43 smithi156 bash[29655]: audit 2023-11-17T06:12:41.360961+0000 mgr.smithi006.odexej (mgr.14176) 1813 : audit [DBG] from='client.15494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:44 smithi006 bash[26022]: cluster 2023-11-17T06:12:42.686693+0000 mgr.smithi006.odexej (mgr.14176) 1814 : cluster [DBG] pgmap v1236: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:44.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:44 smithi156 bash[29655]: cluster 2023-11-17T06:12:42.686693+0000 mgr.smithi006.odexej (mgr.14176) 1814 : cluster [DBG] pgmap v1236: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:46.131 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:46.131 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:46.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:46 smithi006 bash[26022]: cluster 2023-11-17T06:12:44.688153+0000 mgr.smithi006.odexej (mgr.14176) 1815 : cluster [DBG] pgmap v1237: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:46.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:46 smithi156 bash[29655]: cluster 2023-11-17T06:12:44.688153+0000 mgr.smithi006.odexej (mgr.14176) 1815 : cluster [DBG] pgmap v1237: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:46.739 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:47.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:47 smithi006 bash[26022]: audit 2023-11-17T06:12:46.113739+0000 mgr.smithi006.odexej (mgr.14176) 1816 : audit [DBG] from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:47.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:47 smithi006 bash[26022]: audit 2023-11-17T06:12:46.653889+0000 mon.smithi006 (mon.0) 1111 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:12:47.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:47 smithi156 bash[29655]: audit 2023-11-17T06:12:46.113739+0000 mgr.smithi006.odexej (mgr.14176) 1816 : audit [DBG] from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:47.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:47 smithi156 bash[29655]: audit 2023-11-17T06:12:46.653889+0000 mon.smithi006 (mon.0) 1111 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:12:47.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:48.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:48 smithi156 bash[29655]: cluster 2023-11-17T06:12:46.689175+0000 mgr.smithi006.odexej (mgr.14176) 1817 : cluster [DBG] pgmap v1238: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:48.522 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:48 smithi006 bash[26022]: cluster 2023-11-17T06:12:46.689175+0000 mgr.smithi006.odexej (mgr.14176) 1817 : cluster [DBG] pgmap v1238: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:50.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:50 smithi006 bash[26022]: cluster 2023-11-17T06:12:48.690416+0000 mgr.smithi006.odexej (mgr.14176) 1818 : cluster [DBG] pgmap v1239: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:50.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:50 smithi156 bash[29655]: cluster 2023-11-17T06:12:48.690416+0000 mgr.smithi006.odexej (mgr.14176) 1818 : cluster [DBG] pgmap v1239: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:50.849 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:50.849 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:02:22.453068Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:02:21.580302Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:02:22.453586Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:02:21.582135Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:02:21.580609Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:02:21.580814Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:02:22.455691Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:02:21.581008Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:02:21.581200Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:02:22.454093Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:51.475 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:52.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:52 smithi006 bash[26022]: cluster 2023-11-17T06:12:50.691986+0000 mgr.smithi006.odexej (mgr.14176) 1819 : cluster [DBG] pgmap v1240: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:52.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:52 smithi006 bash[26022]: audit 2023-11-17T06:12:50.832808+0000 mgr.smithi006.odexej (mgr.14176) 1820 : audit [DBG] from='client.25241 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:52.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:52 smithi156 bash[29655]: cluster 2023-11-17T06:12:50.691986+0000 mgr.smithi006.odexej (mgr.14176) 1819 : cluster [DBG] pgmap v1240: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:12:52.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:52 smithi156 bash[29655]: audit 2023-11-17T06:12:50.832808+0000 mgr.smithi006.odexej (mgr.14176) 1820 : audit [DBG] from='client.25241 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:52.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:53.623 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:53 smithi156 bash[29655]: audit 2023-11-17T06:12:52.308333+0000 mon.smithi006 (mon.0) 1112 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:53.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:53 smithi006 bash[26022]: audit 2023-11-17T06:12:52.308333+0000 mon.smithi006 (mon.0) 1112 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: cluster 2023-11-17T06:12:52.693091+0000 mgr.smithi006.odexej (mgr.14176) 1821 : cluster [DBG] pgmap v1241: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:53.742457+0000 mon.smithi006 (mon.0) 1113 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:54.036779+0000 mon.smithi006 (mon.0) 1114 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:12:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:54.038197+0000 mon.smithi006 (mon.0) 1115 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:12:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:54.044248+0000 mon.smithi006 (mon.0) 1116 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:54.049219+0000 mon.smithi006 (mon.0) 1117 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:12:54.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:54 smithi006 bash[26022]: audit 2023-11-17T06:12:54.059174+0000 mon.smithi006 (mon.0) 1118 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: cluster 2023-11-17T06:12:52.693091+0000 mgr.smithi006.odexej (mgr.14176) 1821 : cluster [DBG] pgmap v1241: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:53.742457+0000 mon.smithi006 (mon.0) 1113 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:54.036779+0000 mon.smithi006 (mon.0) 1114 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:54.038197+0000 mon.smithi006 (mon.0) 1115 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:54.044248+0000 mon.smithi006 (mon.0) 1116 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:54.049219+0000 mon.smithi006 (mon.0) 1117 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:12:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:54 smithi156 bash[29655]: audit 2023-11-17T06:12:54.059174+0000 mon.smithi006 (mon.0) 1118 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:12:55.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:55 smithi006 bash[26022]: cluster 2023-11-17T06:12:54.045041+0000 mgr.smithi006.odexej (mgr.14176) 1822 : cluster [DBG] pgmap v1242: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:12:55.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:55 smithi006 bash[26022]: cephadm 2023-11-17T06:12:54.062213+0000 mgr.smithi006.odexej (mgr.14176) 1823 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.vbsswv on smithi156 2023-11-17T06:12:55.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:55 smithi006 bash[26022]: cluster 2023-11-17T06:12:55.042388+0000 mon.smithi006 (mon.0) 1119 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:12:55.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:55 smithi006 bash[26022]: cluster 2023-11-17T06:12:55.042472+0000 mon.smithi006 (mon.0) 1120 : cluster [INF] Cluster is now healthy 2023-11-17T06:12:55.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:55 smithi156 bash[29655]: cluster 2023-11-17T06:12:54.045041+0000 mgr.smithi006.odexej (mgr.14176) 1822 : cluster [DBG] pgmap v1242: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:12:55.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:55 smithi156 bash[29655]: cephadm 2023-11-17T06:12:54.062213+0000 mgr.smithi006.odexej (mgr.14176) 1823 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.vbsswv on smithi156 2023-11-17T06:12:55.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:55 smithi156 bash[29655]: cluster 2023-11-17T06:12:55.042388+0000 mon.smithi006 (mon.0) 1119 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:12:55.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:55 smithi156 bash[29655]: cluster 2023-11-17T06:12:55.042472+0000 mon.smithi006 (mon.0) 1120 : cluster [INF] Cluster is now healthy 2023-11-17T06:12:55.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:12:55.843 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:10:28.001863Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wuvwyn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:12:56.479 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:12:57.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:12:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:57 smithi006 bash[26022]: audit 2023-11-17T06:12:55.828926+0000 mgr.smithi006.odexej (mgr.14176) 1824 : audit [DBG] from='client.15506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:57 smithi006 bash[26022]: cluster 2023-11-17T06:12:56.046110+0000 mgr.smithi006.odexej (mgr.14176) 1825 : cluster [DBG] pgmap v1243: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:12:57.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:57 smithi156 bash[29655]: audit 2023-11-17T06:12:55.828926+0000 mgr.smithi006.odexej (mgr.14176) 1824 : audit [DBG] from='client.15506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:12:57.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:57 smithi156 bash[29655]: cluster 2023-11-17T06:12:56.046110+0000 mgr.smithi006.odexej (mgr.14176) 1825 : cluster [DBG] pgmap v1243: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:12:58.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:58 smithi006 bash[26022]: audit 2023-11-17T06:12:57.815863+0000 mon.smithi006 (mon.0) 1121 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.vbsswv"}]: dispatch 2023-11-17T06:12:58.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:58 smithi156 bash[29655]: audit 2023-11-17T06:12:57.815863+0000 mon.smithi006 (mon.0) 1121 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.vbsswv"}]: dispatch 2023-11-17T06:12:59.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: cephadm 2023-11-17T06:12:57.814671+0000 mgr.smithi006.odexej (mgr.14176) 1826 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:59.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.671 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: cephadm 2023-11-17T06:12:57.815175+0000 mgr.smithi006.odexej (mgr.14176) 1827 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: cephadm 2023-11-17T06:12:57.817639+0000 mgr.smithi006.odexej (mgr.14176) 1828 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.672 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.673 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: cephadm 2023-11-17T06:12:57.822641+0000 mgr.smithi006.odexej (mgr.14176) 1829 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.gudaaa on smithi006 2023-11-17T06:12:59.674 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:12:59 smithi006 bash[26022]: cluster 2023-11-17T06:12:58.046929+0000 mgr.smithi006.odexej (mgr.14176) 1830 : cluster [DBG] pgmap v1244: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: cephadm 2023-11-17T06:12:57.814671+0000 mgr.smithi006.odexej (mgr.14176) 1826 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:12:59.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: cephadm 2023-11-17T06:12:57.815175+0000 mgr.smithi006.odexej (mgr.14176) 1827 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.686 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: cephadm 2023-11-17T06:12:57.817639+0000 mgr.smithi006.odexej (mgr.14176) 1828 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv 2023-11-17T06:12:59.687 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.vbsswv ... 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:12:59.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: cephadm 2023-11-17T06:12:57.822641+0000 mgr.smithi006.odexej (mgr.14176) 1829 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.gudaaa on smithi006 2023-11-17T06:12:59.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:12:59 smithi156 bash[29655]: cluster 2023-11-17T06:12:58.046929+0000 mgr.smithi006.odexej (mgr.14176) 1830 : cluster [DBG] pgmap v1244: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:13:00.677 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:00 smithi006 bash[26022]: cluster 2023-11-17T06:13:00.048124+0000 mgr.smithi006.odexej (mgr.14176) 1831 : cluster [DBG] pgmap v1245: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:13:00.810 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:00 smithi156 bash[29655]: cluster 2023-11-17T06:13:00.048124+0000 mgr.smithi006.odexej (mgr.14176) 1831 : cluster [DBG] pgmap v1245: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2023-11-17T06:13:00.810 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:00.810 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:42.902839Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.cjxqzl on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-cjxqzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.cjxqzl\nDeploy daemon haproxy.nfs.foo.smithi156.cjxqzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.641728Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.aneiee on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-aneiee\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.aneiee\nDeploy daemon haproxy.nfs.foo.smithi006.aneiee ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:11:46.646016Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.bchrqb on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:01.459 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:01.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:01 smithi006 bash[26022]: audit 2023-11-17T06:13:00.796590+0000 mgr.smithi006.odexej (mgr.14176) 1832 : audit [DBG] from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:01.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:01 smithi156 bash[29655]: audit 2023-11-17T06:13:00.796590+0000 mgr.smithi006.odexej (mgr.14176) 1832 : audit [DBG] from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:02.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:02.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cephadm 2023-11-17T06:13:01.801454+0000 mgr.smithi006.odexej (mgr.14176) 1833 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cephadm 2023-11-17T06:13:01.801821+0000 mgr.smithi006.odexej (mgr.14176) 1834 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: audit 2023-11-17T06:13:01.802489+0000 mon.smithi006 (mon.0) 1122 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.gudaaa"}]: dispatch 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cephadm 2023-11-17T06:13:01.803733+0000 mgr.smithi006.odexej (mgr.14176) 1835 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.832 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cephadm 2023-11-17T06:13:01.808280+0000 mgr.smithi006.odexej (mgr.14176) 1836 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cephadm 2023-11-17T06:13:01.812101+0000 mgr.smithi006.odexej (mgr.14176) 1837 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:13:02.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:02 smithi006 bash[26022]: cluster 2023-11-17T06:13:01.813352+0000 mgr.smithi006.odexej (mgr.14176) 1838 : cluster [DBG] pgmap v1246: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:13:02.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cephadm 2023-11-17T06:13:01.801454+0000 mgr.smithi006.odexej (mgr.14176) 1833 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.834 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.883 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.884 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cephadm 2023-11-17T06:13:01.801821+0000 mgr.smithi006.odexej (mgr.14176) 1834 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: audit 2023-11-17T06:13:01.802489+0000 mon.smithi006 (mon.0) 1122 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.gudaaa"}]: dispatch 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cephadm 2023-11-17T06:13:01.803733+0000 mgr.smithi006.odexej (mgr.14176) 1835 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:13:02.885 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.gudaaa ... 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cephadm 2023-11-17T06:13:01.808280+0000 mgr.smithi006.odexej (mgr.14176) 1836 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cephadm 2023-11-17T06:13:01.812101+0000 mgr.smithi006.odexej (mgr.14176) 1837 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:13:02.886 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:02 smithi156 bash[29655]: cluster 2023-11-17T06:13:01.813352+0000 mgr.smithi006.odexej (mgr.14176) 1838 : cluster [DBG] pgmap v1246: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:13:03.623 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:03 smithi156 bash[29655]: cluster 2023-11-17T06:13:02.352004+0000 mon.smithi006 (mon.0) 1123 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:13:03.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:03 smithi006 bash[26022]: cluster 2023-11-17T06:13:02.352004+0000 mon.smithi006 (mon.0) 1123 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:13:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:04 smithi006 bash[26022]: cluster 2023-11-17T06:13:03.814494+0000 mgr.smithi006.odexej (mgr.14176) 1839 : cluster [DBG] pgmap v1247: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:13:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:04 smithi156 bash[29655]: cluster 2023-11-17T06:13:03.814494+0000 mgr.smithi006.odexej (mgr.14176) 1839 : cluster [DBG] pgmap v1247: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2023-11-17T06:13:05.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:05.705 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:06.260 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:07.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:06 smithi006 bash[26022]: audit 2023-11-17T06:13:05.695453+0000 mgr.smithi006.odexej (mgr.14176) 1840 : audit [DBG] from='client.15514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:07.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:06 smithi006 bash[26022]: cluster 2023-11-17T06:13:05.815633+0000 mgr.smithi006.odexej (mgr.14176) 1841 : cluster [DBG] pgmap v1248: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:07.166 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:06 smithi006 bash[26022]: audit 2023-11-17T06:13:06.546401+0000 mon.smithi006 (mon.0) 1124 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:13:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:06 smithi156 bash[29655]: audit 2023-11-17T06:13:05.695453+0000 mgr.smithi006.odexej (mgr.14176) 1840 : audit [DBG] from='client.15514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:06 smithi156 bash[29655]: cluster 2023-11-17T06:13:05.815633+0000 mgr.smithi006.odexej (mgr.14176) 1841 : cluster [DBG] pgmap v1248: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:07.181 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:06 smithi156 bash[29655]: audit 2023-11-17T06:13:06.546401+0000 mon.smithi006 (mon.0) 1124 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:13:07.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:09.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:08 smithi006 bash[26022]: cluster 2023-11-17T06:13:07.816639+0000 mgr.smithi006.odexej (mgr.14176) 1842 : cluster [DBG] pgmap v1249: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:09.436 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:09 smithi156 bash[29655]: cluster 2023-11-17T06:13:07.816639+0000 mgr.smithi006.odexej (mgr.14176) 1842 : cluster [DBG] pgmap v1249: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:10.395 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:10.396 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:10.962 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:11 smithi006 bash[26022]: cluster 2023-11-17T06:13:09.818056+0000 mgr.smithi006.odexej (mgr.14176) 1843 : cluster [DBG] pgmap v1250: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:11.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:11 smithi156 bash[29655]: cluster 2023-11-17T06:13:09.818056+0000 mgr.smithi006.odexej (mgr.14176) 1843 : cluster [DBG] pgmap v1250: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:11.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:12 smithi006 bash[26022]: audit 2023-11-17T06:13:10.386219+0000 mgr.smithi006.odexej (mgr.14176) 1844 : audit [DBG] from='client.15518 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:12.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:12 smithi156 bash[29655]: audit 2023-11-17T06:13:10.386219+0000 mgr.smithi006.odexej (mgr.14176) 1844 : audit [DBG] from='client.15518 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:13.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:13 smithi006 bash[26022]: cluster 2023-11-17T06:13:11.819049+0000 mgr.smithi006.odexej (mgr.14176) 1845 : cluster [DBG] pgmap v1251: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:13.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:13 smithi156 bash[29655]: cluster 2023-11-17T06:13:11.819049+0000 mgr.smithi006.odexej (mgr.14176) 1845 : cluster [DBG] pgmap v1251: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:13:15.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:15.074 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:15.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:15 smithi006 bash[26022]: cluster 2023-11-17T06:13:13.820535+0000 mgr.smithi006.odexej (mgr.14176) 1846 : cluster [DBG] pgmap v1252: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:15.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:15 smithi156 bash[29655]: cluster 2023-11-17T06:13:13.820535+0000 mgr.smithi006.odexej (mgr.14176) 1846 : cluster [DBG] pgmap v1252: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:15.685 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:16.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:16 smithi006 bash[26022]: audit 2023-11-17T06:13:15.058265+0000 mgr.smithi006.odexej (mgr.14176) 1847 : audit [DBG] from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:16.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:16 smithi156 bash[29655]: audit 2023-11-17T06:13:15.058265+0000 mgr.smithi006.odexej (mgr.14176) 1847 : audit [DBG] from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:16.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:17.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:17 smithi006 bash[26022]: cluster 2023-11-17T06:13:15.822020+0000 mgr.smithi006.odexej (mgr.14176) 1848 : cluster [DBG] pgmap v1253: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:17.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:17 smithi156 bash[29655]: cluster 2023-11-17T06:13:15.822020+0000 mgr.smithi006.odexej (mgr.14176) 1848 : cluster [DBG] pgmap v1253: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:19.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:19 smithi006 bash[26022]: cluster 2023-11-17T06:13:17.823007+0000 mgr.smithi006.odexej (mgr.14176) 1849 : cluster [DBG] pgmap v1254: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:19.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:19 smithi156 bash[29655]: cluster 2023-11-17T06:13:17.823007+0000 mgr.smithi006.odexej (mgr.14176) 1849 : cluster [DBG] pgmap v1254: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:19.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:19.697 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:20.302 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:21.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:21.315 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:21 smithi006 bash[26022]: audit 2023-11-17T06:13:19.680870+0000 mgr.smithi006.odexej (mgr.14176) 1850 : audit [DBG] from='client.15526 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:21.315 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:21 smithi006 bash[26022]: cluster 2023-11-17T06:13:19.824405+0000 mgr.smithi006.odexej (mgr.14176) 1851 : cluster [DBG] pgmap v1255: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:21.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:21 smithi156 bash[29655]: audit 2023-11-17T06:13:19.680870+0000 mgr.smithi006.odexej (mgr.14176) 1850 : audit [DBG] from='client.15526 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:21.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:21 smithi156 bash[29655]: cluster 2023-11-17T06:13:19.824405+0000 mgr.smithi006.odexej (mgr.14176) 1851 : cluster [DBG] pgmap v1255: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:23.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:23 smithi006 bash[26022]: cluster 2023-11-17T06:13:21.825443+0000 mgr.smithi006.odexej (mgr.14176) 1852 : cluster [DBG] pgmap v1256: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:23.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:23 smithi156 bash[29655]: cluster 2023-11-17T06:13:21.825443+0000 mgr.smithi006.odexej (mgr.14176) 1852 : cluster [DBG] pgmap v1256: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:24.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:24.512 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:25.146 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:25.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:25 smithi006 bash[26022]: cluster 2023-11-17T06:13:23.826647+0000 mgr.smithi006.odexej (mgr.14176) 1853 : cluster [DBG] pgmap v1257: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:25.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:25 smithi156 bash[29655]: cluster 2023-11-17T06:13:23.826647+0000 mgr.smithi006.odexej (mgr.14176) 1853 : cluster [DBG] pgmap v1257: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:26.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:26.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:26 smithi006 bash[26022]: audit 2023-11-17T06:13:24.490101+0000 mgr.smithi006.odexej (mgr.14176) 1854 : audit [DBG] from='client.15530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:26.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:26 smithi156 bash[29655]: audit 2023-11-17T06:13:24.490101+0000 mgr.smithi006.odexej (mgr.14176) 1854 : audit [DBG] from='client.15530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:27.419 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:27 smithi006 bash[26022]: cluster 2023-11-17T06:13:25.828002+0000 mgr.smithi006.odexej (mgr.14176) 1855 : cluster [DBG] pgmap v1258: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:27.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:27 smithi156 bash[29655]: cluster 2023-11-17T06:13:25.828002+0000 mgr.smithi006.odexej (mgr.14176) 1855 : cluster [DBG] pgmap v1258: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:29.168 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:29.168 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:29.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:29 smithi006 bash[26022]: cluster 2023-11-17T06:13:27.829078+0000 mgr.smithi006.odexej (mgr.14176) 1856 : cluster [DBG] pgmap v1259: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:29.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:29 smithi156 bash[29655]: cluster 2023-11-17T06:13:27.829078+0000 mgr.smithi006.odexej (mgr.14176) 1856 : cluster [DBG] pgmap v1259: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:29.774 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:30.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:30 smithi006 bash[26022]: audit 2023-11-17T06:13:29.150977+0000 mgr.smithi006.odexej (mgr.14176) 1857 : audit [DBG] from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:30.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:30 smithi156 bash[29655]: audit 2023-11-17T06:13:29.150977+0000 mgr.smithi006.odexej (mgr.14176) 1857 : audit [DBG] from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:30.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:31.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:31 smithi006 bash[26022]: cluster 2023-11-17T06:13:29.830465+0000 mgr.smithi006.odexej (mgr.14176) 1858 : cluster [DBG] pgmap v1260: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:31.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:31 smithi156 bash[29655]: cluster 2023-11-17T06:13:29.830465+0000 mgr.smithi006.odexej (mgr.14176) 1858 : cluster [DBG] pgmap v1260: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:33.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:33 smithi006 bash[26022]: cluster 2023-11-17T06:13:31.831546+0000 mgr.smithi006.odexej (mgr.14176) 1859 : cluster [DBG] pgmap v1261: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:33.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:33 smithi156 bash[29655]: cluster 2023-11-17T06:13:31.831546+0000 mgr.smithi006.odexej (mgr.14176) 1859 : cluster [DBG] pgmap v1261: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:33.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:33.861 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:34.468 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:35.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:35 smithi006 bash[26022]: cluster 2023-11-17T06:13:33.832683+0000 mgr.smithi006.odexej (mgr.14176) 1860 : cluster [DBG] pgmap v1262: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:35.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:35 smithi006 bash[26022]: audit 2023-11-17T06:13:33.843582+0000 mgr.smithi006.odexej (mgr.14176) 1861 : audit [DBG] from='client.15538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:35.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:35 smithi156 bash[29655]: cluster 2023-11-17T06:13:33.832683+0000 mgr.smithi006.odexej (mgr.14176) 1860 : cluster [DBG] pgmap v1262: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:35.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:35 smithi156 bash[29655]: audit 2023-11-17T06:13:33.843582+0000 mgr.smithi006.odexej (mgr.14176) 1861 : audit [DBG] from='client.15538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:35.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:37.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:37 smithi006 bash[26022]: cluster 2023-11-17T06:13:35.834095+0000 mgr.smithi006.odexej (mgr.14176) 1862 : cluster [DBG] pgmap v1263: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:37.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:37 smithi156 bash[29655]: cluster 2023-11-17T06:13:35.834095+0000 mgr.smithi006.odexej (mgr.14176) 1862 : cluster [DBG] pgmap v1263: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:38.591 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:38.591 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:39.199 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:39.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:39 smithi006 bash[26022]: cluster 2023-11-17T06:13:37.835191+0000 mgr.smithi006.odexej (mgr.14176) 1863 : cluster [DBG] pgmap v1264: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:39.435 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:39 smithi156 bash[29655]: cluster 2023-11-17T06:13:37.835191+0000 mgr.smithi006.odexej (mgr.14176) 1863 : cluster [DBG] pgmap v1264: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:40.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:40.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:40 smithi006 bash[26022]: audit 2023-11-17T06:13:38.581476+0000 mgr.smithi006.odexej (mgr.14176) 1864 : audit [DBG] from='client.15542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:40.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:40 smithi156 bash[29655]: audit 2023-11-17T06:13:38.581476+0000 mgr.smithi006.odexej (mgr.14176) 1864 : audit [DBG] from='client.15542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:41.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:41 smithi006 bash[26022]: cluster 2023-11-17T06:13:39.836602+0000 mgr.smithi006.odexej (mgr.14176) 1865 : cluster [DBG] pgmap v1265: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:41.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:41 smithi156 bash[29655]: cluster 2023-11-17T06:13:39.836602+0000 mgr.smithi006.odexej (mgr.14176) 1865 : cluster [DBG] pgmap v1265: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:43.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:43.195 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:43.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:43 smithi006 bash[26022]: cluster 2023-11-17T06:13:41.837600+0000 mgr.smithi006.odexej (mgr.14176) 1866 : cluster [DBG] pgmap v1266: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:43.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:43 smithi156 bash[29655]: cluster 2023-11-17T06:13:41.837600+0000 mgr.smithi006.odexej (mgr.14176) 1866 : cluster [DBG] pgmap v1266: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:43.767 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:44.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:44 smithi006 bash[26022]: audit 2023-11-17T06:13:43.179230+0000 mgr.smithi006.odexej (mgr.14176) 1867 : audit [DBG] from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:44.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:44 smithi156 bash[29655]: audit 2023-11-17T06:13:43.179230+0000 mgr.smithi006.odexej (mgr.14176) 1867 : audit [DBG] from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:44.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:45.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:45 smithi156 bash[29655]: cluster 2023-11-17T06:13:43.838796+0000 mgr.smithi006.odexej (mgr.14176) 1868 : cluster [DBG] pgmap v1267: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:45.509 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:45 smithi006 bash[26022]: cluster 2023-11-17T06:13:43.838796+0000 mgr.smithi006.odexej (mgr.14176) 1868 : cluster [DBG] pgmap v1267: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:47.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:47 smithi006 bash[26022]: cluster 2023-11-17T06:13:45.840230+0000 mgr.smithi006.odexej (mgr.14176) 1869 : cluster [DBG] pgmap v1268: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:47.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:47 smithi156 bash[29655]: cluster 2023-11-17T06:13:45.840230+0000 mgr.smithi006.odexej (mgr.14176) 1869 : cluster [DBG] pgmap v1268: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:47.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:47.818 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:48.386 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:49.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:49.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:49 smithi006 bash[26022]: audit 2023-11-17T06:13:47.808984+0000 mgr.smithi006.odexej (mgr.14176) 1870 : audit [DBG] from='client.15550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:49.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:49 smithi006 bash[26022]: cluster 2023-11-17T06:13:47.841316+0000 mgr.smithi006.odexej (mgr.14176) 1871 : cluster [DBG] pgmap v1269: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:49.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:49 smithi156 bash[29655]: audit 2023-11-17T06:13:47.808984+0000 mgr.smithi006.odexej (mgr.14176) 1870 : audit [DBG] from='client.15550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:49.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:49 smithi156 bash[29655]: cluster 2023-11-17T06:13:47.841316+0000 mgr.smithi006.odexej (mgr.14176) 1871 : cluster [DBG] pgmap v1269: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:51.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:51 smithi156 bash[29655]: cluster 2023-11-17T06:13:49.842693+0000 mgr.smithi006.odexej (mgr.14176) 1872 : cluster [DBG] pgmap v1270: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:51.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:51 smithi006 bash[26022]: cluster 2023-11-17T06:13:49.842693+0000 mgr.smithi006.odexej (mgr.14176) 1872 : cluster [DBG] pgmap v1270: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:52.593 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:52.594 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:53.188 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:53.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:53 smithi156 bash[29655]: cluster 2023-11-17T06:13:51.843763+0000 mgr.smithi006.odexej (mgr.14176) 1873 : cluster [DBG] pgmap v1271: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:53.501 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:53 smithi006 bash[26022]: cluster 2023-11-17T06:13:51.843763+0000 mgr.smithi006.odexej (mgr.14176) 1873 : cluster [DBG] pgmap v1271: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:54.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:54.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:54 smithi006 bash[26022]: audit 2023-11-17T06:13:52.584438+0000 mgr.smithi006.odexej (mgr.14176) 1874 : audit [DBG] from='client.15554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:54.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:54 smithi156 bash[29655]: audit 2023-11-17T06:13:52.584438+0000 mgr.smithi006.odexej (mgr.14176) 1874 : audit [DBG] from='client.15554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:55.617 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:55 smithi006 bash[26022]: cluster 2023-11-17T06:13:53.845002+0000 mgr.smithi006.odexej (mgr.14176) 1875 : cluster [DBG] pgmap v1272: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:55.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:55 smithi156 bash[29655]: cluster 2023-11-17T06:13:53.845002+0000 mgr.smithi006.odexej (mgr.14176) 1875 : cluster [DBG] pgmap v1272: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:13:57.294 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:13:57.294 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:13:57.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:57 smithi006 bash[26022]: cluster 2023-11-17T06:13:55.846453+0000 mgr.smithi006.odexej (mgr.14176) 1876 : cluster [DBG] pgmap v1273: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:57.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:57 smithi156 bash[29655]: cluster 2023-11-17T06:13:55.846453+0000 mgr.smithi006.odexej (mgr.14176) 1876 : cluster [DBG] pgmap v1273: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:57.919 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:13:58.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:58 smithi006 bash[26022]: audit 2023-11-17T06:13:57.279925+0000 mgr.smithi006.odexej (mgr.14176) 1877 : audit [DBG] from='client.15558 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:58.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:58 smithi156 bash[29655]: audit 2023-11-17T06:13:57.279925+0000 mgr.smithi006.odexej (mgr.14176) 1877 : audit [DBG] from='client.15558 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:13:58.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:13:59.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:13:59 smithi006 bash[26022]: cluster 2023-11-17T06:13:57.847561+0000 mgr.smithi006.odexej (mgr.14176) 1878 : cluster [DBG] pgmap v1274: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:13:59.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:13:59 smithi156 bash[29655]: cluster 2023-11-17T06:13:57.847561+0000 mgr.smithi006.odexej (mgr.14176) 1878 : cluster [DBG] pgmap v1274: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:14:01.559 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:01 smithi006 bash[26022]: cluster 2023-11-17T06:13:59.848959+0000 mgr.smithi006.odexej (mgr.14176) 1879 : cluster [DBG] pgmap v1275: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:14:03.785 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:14:03.786 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:11:46.650313Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.bhnwaf on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:14:03.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:02 smithi006 bash[26022]: audit 2023-11-17T06:14:01.817674+0000 mon.smithi006 (mon.0) 1125 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:14:03.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:02 smithi006 bash[26022]: audit 2023-11-17T06:14:02.123996+0000 mon.smithi006 (mon.0) 1126 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:03.786 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:01 smithi156 bash[29655]: cluster 2023-11-17T06:13:59.848959+0000 mgr.smithi006.odexej (mgr.14176) 1879 : cluster [DBG] pgmap v1275: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:14:03.786 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:02 smithi156 bash[29655]: audit 2023-11-17T06:14:01.817674+0000 mon.smithi006 (mon.0) 1125 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:14:03.786 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:02 smithi156 bash[29655]: audit 2023-11-17T06:14:02.123996+0000 mon.smithi006 (mon.0) 1126 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: cluster 2023-11-17T06:14:01.849686+0000 mgr.smithi006.odexej (mgr.14176) 1880 : cluster [DBG] pgmap v1276: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:01.907335+0000 mgr.smithi006.odexej (mgr.14176) 1881 : audit [DBG] from='client.15562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:02.419971+0000 mon.smithi006 (mon.0) 1127 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:02.421524+0000 mon.smithi006 (mon.0) 1128 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:02.430266+0000 mon.smithi006 (mon.0) 1129 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:02.440054+0000 mon.smithi006 (mon.0) 1130 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:14:03.787 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:03 smithi156 bash[29655]: audit 2023-11-17T06:14:02.453738+0000 mon.smithi006 (mon.0) 1131 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:03.788 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:14:03.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: cluster 2023-11-17T06:14:01.849686+0000 mgr.smithi006.odexej (mgr.14176) 1880 : cluster [DBG] pgmap v1276: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:01.907335+0000 mgr.smithi006.odexej (mgr.14176) 1881 : audit [DBG] from='client.15562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:02.419971+0000 mon.smithi006 (mon.0) 1127 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:02.421524+0000 mon.smithi006 (mon.0) 1128 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:02.430266+0000 mon.smithi006 (mon.0) 1129 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:02.440054+0000 mon.smithi006 (mon.0) 1130 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:14:03.789 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:03 smithi006 bash[26022]: audit 2023-11-17T06:14:02.453738+0000 mon.smithi006 (mon.0) 1131 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:04 smithi006 bash[26022]: cluster 2023-11-17T06:14:02.431261+0000 mgr.smithi006.odexej (mgr.14176) 1882 : cluster [DBG] pgmap v1277: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:14:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:04 smithi006 bash[26022]: cephadm 2023-11-17T06:14:02.457364+0000 mgr.smithi006.odexej (mgr.14176) 1883 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.oikrsx on smithi156 2023-11-17T06:14:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:04 smithi006 bash[26022]: cluster 2023-11-17T06:14:03.426983+0000 mon.smithi006 (mon.0) 1132 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:14:04.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:04 smithi006 bash[26022]: cluster 2023-11-17T06:14:03.427054+0000 mon.smithi006 (mon.0) 1133 : cluster [INF] Cluster is now healthy 2023-11-17T06:14:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:04 smithi156 bash[29655]: cluster 2023-11-17T06:14:02.431261+0000 mgr.smithi006.odexej (mgr.14176) 1882 : cluster [DBG] pgmap v1277: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:14:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:04 smithi156 bash[29655]: cephadm 2023-11-17T06:14:02.457364+0000 mgr.smithi006.odexej (mgr.14176) 1883 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.oikrsx on smithi156 2023-11-17T06:14:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:04 smithi156 bash[29655]: cluster 2023-11-17T06:14:03.426983+0000 mon.smithi006 (mon.0) 1132 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:14:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:04 smithi156 bash[29655]: cluster 2023-11-17T06:14:03.427054+0000 mon.smithi006 (mon.0) 1133 : cluster [INF] Cluster is now healthy 2023-11-17T06:14:04.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:14:06.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:05 smithi156 bash[29655]: cluster 2023-11-17T06:14:04.432403+0000 mgr.smithi006.odexej (mgr.14176) 1884 : cluster [DBG] pgmap v1278: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:06.147 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:05 smithi156 bash[29655]: audit 2023-11-17T06:14:04.752524+0000 mon.smithi006 (mon.0) 1134 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:06.148 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:05 smithi006 bash[26022]: cluster 2023-11-17T06:14:04.432403+0000 mgr.smithi006.odexej (mgr.14176) 1884 : cluster [DBG] pgmap v1278: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:06.148 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:05 smithi006 bash[26022]: audit 2023-11-17T06:14:04.752524+0000 mon.smithi006 (mon.0) 1134 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:07.175 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: cephadm 2023-11-17T06:14:06.208414+0000 mgr.smithi006.odexej (mgr.14176) 1885 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.197 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.197 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.197 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.204 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.212 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.214 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.216 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.217 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.217 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:14:07.219 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:14:07.220 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:14:07.222 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:14:07.224 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.225 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: cephadm 2023-11-17T06:14:06.209087+0000 mgr.smithi006.odexej (mgr.14176) 1886 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: audit 2023-11-17T06:14:06.209836+0000 mon.smithi006 (mon.0) 1135 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.oikrsx"}]: dispatch 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: cephadm 2023-11-17T06:14:06.212096+0000 mgr.smithi006.odexej (mgr.14176) 1887 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.226 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.227 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:06 smithi006 bash[26022]: cephadm 2023-11-17T06:14:06.217159+0000 mgr.smithi006.odexej (mgr.14176) 1888 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.iaeoyq on smithi006 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: cephadm 2023-11-17T06:14:06.208414+0000 mgr.smithi006.odexej (mgr.14176) 1885 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.228 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.229 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.230 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.231 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: cephadm 2023-11-17T06:14:06.209087+0000 mgr.smithi006.odexej (mgr.14176) 1886 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.231 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: audit 2023-11-17T06:14:06.209836+0000 mon.smithi006 (mon.0) 1135 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.oikrsx"}]: dispatch 2023-11-17T06:14:07.231 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: cephadm 2023-11-17T06:14:06.212096+0000 mgr.smithi006.odexej (mgr.14176) 1887 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.231 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.oikrsx ... 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:07.232 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:06 smithi156 bash[29655]: cephadm 2023-11-17T06:14:06.217159+0000 mgr.smithi006.odexej (mgr.14176) 1888 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.iaeoyq on smithi006 2023-11-17T06:14:08.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:14:08.031 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:12:57.817328Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.vbsswv on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-vbsswv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.vbsswv\nDeploy daemon haproxy.nfs.foo.smithi156.vbsswv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.803585Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.gudaaa on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-gudaaa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.gudaaa\nDeploy daemon haproxy.nfs.foo.smithi006.gudaaa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:13:01.808056Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.jtrsry on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:06.211705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\nDeploy daemon haproxy.nfs.foo.smithi156.oikrsx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:14:08.418 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:08 smithi006 bash[26022]: cluster 2023-11-17T06:14:06.433009+0000 mgr.smithi006.odexej (mgr.14176) 1889 : cluster [DBG] pgmap v1279: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:08.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:08 smithi156 bash[29655]: cluster 2023-11-17T06:14:06.433009+0000 mgr.smithi006.odexej (mgr.14176) 1889 : cluster [DBG] pgmap v1279: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:08.613 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:14:09.426 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:09 smithi006 bash[26022]: audit 2023-11-17T06:14:08.017051+0000 mgr.smithi006.odexej (mgr.14176) 1890 : audit [DBG] from='client.15566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:14:09.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:09 smithi156 bash[29655]: audit 2023-11-17T06:14:08.017051+0000 mgr.smithi006.odexej (mgr.14176) 1890 : audit [DBG] from='client.15566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:14:09.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:14:10.378 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:10 smithi006 bash[26022]: cluster 2023-11-17T06:14:08.433856+0000 mgr.smithi006.odexej (mgr.14176) 1891 : cluster [DBG] pgmap v1280: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:10.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:10 smithi156 bash[29655]: cluster 2023-11-17T06:14:08.433856+0000 mgr.smithi006.odexej (mgr.14176) 1891 : cluster [DBG] pgmap v1280: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:11 smithi006 bash[26022]: audit 2023-11-17T06:14:10.657483+0000 mon.smithi006 (mon.0) 1136 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.iaeoyq"}]: dispatch 2023-11-17T06:14:11.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:11 smithi156 bash[29655]: audit 2023-11-17T06:14:10.657483+0000 mon.smithi006 (mon.0) 1136 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.iaeoyq"}]: dispatch 2023-11-17T06:14:12.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cluster 2023-11-17T06:14:10.434959+0000 mgr.smithi006.odexej (mgr.14176) 1892 : cluster [DBG] pgmap v1281: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:12.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cephadm 2023-11-17T06:14:10.656622+0000 mgr.smithi006.odexej (mgr.14176) 1893 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:14:12.421 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.422 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cephadm 2023-11-17T06:14:10.656910+0000 mgr.smithi006.odexej (mgr.14176) 1894 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.434 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cephadm 2023-11-17T06:14:10.658793+0000 mgr.smithi006.odexej (mgr.14176) 1895 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.435 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.436 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cephadm 2023-11-17T06:14:10.660702+0000 mgr.smithi006.odexej (mgr.14176) 1896 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:14:12.436 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cephadm 2023-11-17T06:14:10.662328+0000 mgr.smithi006.odexej (mgr.14176) 1897 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:14:12.436 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cluster 2023-11-17T06:14:10.663592+0000 mgr.smithi006.odexej (mgr.14176) 1898 : cluster [DBG] pgmap v1282: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 116 B/s rd, 0 op/s 2023-11-17T06:14:12.436 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: cluster 2023-11-17T06:14:11.023174+0000 mon.smithi006 (mon.0) 1137 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:14:12.436 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:12 smithi006 bash[26022]: audit 2023-11-17T06:14:11.599864+0000 mon.smithi006 (mon.0) 1138 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cluster 2023-11-17T06:14:10.434959+0000 mgr.smithi006.odexej (mgr.14176) 1892 : cluster [DBG] pgmap v1281: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cephadm 2023-11-17T06:14:10.656622+0000 mgr.smithi006.odexej (mgr.14176) 1893 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.437 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:14:12.438 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cephadm 2023-11-17T06:14:10.656910+0000 mgr.smithi006.odexej (mgr.14176) 1894 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cephadm 2023-11-17T06:14:10.658793+0000 mgr.smithi006.odexej (mgr.14176) 1895 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.iaeoyq ... 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:14:12.440 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cephadm 2023-11-17T06:14:10.660702+0000 mgr.smithi006.odexej (mgr.14176) 1896 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cephadm 2023-11-17T06:14:10.662328+0000 mgr.smithi006.odexej (mgr.14176) 1897 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cluster 2023-11-17T06:14:10.663592+0000 mgr.smithi006.odexej (mgr.14176) 1898 : cluster [DBG] pgmap v1282: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 116 B/s rd, 0 op/s 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: cluster 2023-11-17T06:14:11.023174+0000 mon.smithi006 (mon.0) 1137 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:14:12.441 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:12 smithi156 bash[29655]: audit 2023-11-17T06:14:11.599864+0000 mon.smithi006 (mon.0) 1138 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:14:13.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:14:13.151 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:06.211705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\nDeploy daemon haproxy.nfs.foo.smithi156.oikrsx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.658669Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\nDeploy daemon haproxy.nfs.foo.smithi006.iaeoyq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.660594Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:14:13.730 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:14:14.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:14 smithi006 bash[26022]: cluster 2023-11-17T06:14:12.664691+0000 mgr.smithi006.odexej (mgr.14176) 1899 : cluster [DBG] pgmap v1283: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:14 smithi006 bash[26022]: audit 2023-11-17T06:14:13.135051+0000 mgr.smithi006.odexej (mgr.14176) 1900 : audit [DBG] from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:03.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:03.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:16 smithi006 bash[26022]: cluster 2023-11-17T06:14:14.666119+0000 mgr.smithi006.odexej (mgr.14176) 1901 : cluster [DBG] pgmap v1284: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:18 smithi006 bash[26022]: cluster 2023-11-17T06:14:16.667148+0000 mgr.smithi006.odexej (mgr.14176) 1902 : cluster [DBG] pgmap v1285: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:20 smithi006 bash[26022]: cluster 2023-11-17T06:14:18.668324+0000 mgr.smithi006.odexej (mgr.14176) 1903 : cluster [DBG] pgmap v1286: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:20 smithi006 bash[26022]: audit 2023-11-17T06:14:19.755134+0000 mon.smithi006 (mon.0) 1139 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:22 smithi006 bash[26022]: cluster 2023-11-17T06:14:20.669634+0000 mgr.smithi006.odexej (mgr.14176) 1904 : cluster [DBG] pgmap v1287: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:24 smithi006 bash[26022]: cluster 2023-11-17T06:14:22.670672+0000 mgr.smithi006.odexej (mgr.14176) 1905 : cluster [DBG] pgmap v1288: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:26 smithi006 bash[26022]: cluster 2023-11-17T06:14:24.672132+0000 mgr.smithi006.odexej (mgr.14176) 1906 : cluster [DBG] pgmap v1289: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:28 smithi006 bash[26022]: cluster 2023-11-17T06:14:26.673152+0000 mgr.smithi006.odexej (mgr.14176) 1907 : cluster [DBG] pgmap v1290: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:30 smithi006 bash[26022]: cluster 2023-11-17T06:14:28.674256+0000 mgr.smithi006.odexej (mgr.14176) 1908 : cluster [DBG] pgmap v1291: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:32 smithi006 bash[26022]: cluster 2023-11-17T06:14:30.675624+0000 mgr.smithi006.odexej (mgr.14176) 1909 : cluster [DBG] pgmap v1292: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:34 smithi006 bash[26022]: cluster 2023-11-17T06:14:32.676729+0000 mgr.smithi006.odexej (mgr.14176) 1910 : cluster [DBG] pgmap v1293: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:36 smithi006 bash[26022]: cluster 2023-11-17T06:14:34.678101+0000 mgr.smithi006.odexej (mgr.14176) 1911 : cluster [DBG] pgmap v1294: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:38 smithi006 bash[26022]: cluster 2023-11-17T06:14:36.679119+0000 mgr.smithi006.odexej (mgr.14176) 1912 : cluster [DBG] pgmap v1295: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:40 smithi006 bash[26022]: cluster 2023-11-17T06:14:38.680567+0000 mgr.smithi006.odexej (mgr.14176) 1913 : cluster [DBG] pgmap v1296: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:42 smithi006 bash[26022]: cluster 2023-11-17T06:14:40.682101+0000 mgr.smithi006.odexej (mgr.14176) 1914 : cluster [DBG] pgmap v1297: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:44 smithi006 bash[26022]: cluster 2023-11-17T06:14:42.683178+0000 mgr.smithi006.odexej (mgr.14176) 1915 : cluster [DBG] pgmap v1298: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:46 smithi006 bash[26022]: cluster 2023-11-17T06:14:44.684626+0000 mgr.smithi006.odexej (mgr.14176) 1916 : cluster [DBG] pgmap v1299: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:48 smithi006 bash[26022]: cluster 2023-11-17T06:14:46.685797+0000 mgr.smithi006.odexej (mgr.14176) 1917 : cluster [DBG] pgmap v1300: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:50 smithi006 bash[26022]: cluster 2023-11-17T06:14:48.686955+0000 mgr.smithi006.odexej (mgr.14176) 1918 : cluster [DBG] pgmap v1301: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:52 smithi006 bash[26022]: cluster 2023-11-17T06:14:50.688296+0000 mgr.smithi006.odexej (mgr.14176) 1919 : cluster [DBG] pgmap v1302: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:54 smithi006 bash[26022]: cluster 2023-11-17T06:14:52.689314+0000 mgr.smithi006.odexej (mgr.14176) 1920 : cluster [DBG] pgmap v1303: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:56 smithi006 bash[26022]: cluster 2023-11-17T06:14:54.690751+0000 mgr.smithi006.odexej (mgr.14176) 1921 : cluster [DBG] pgmap v1304: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:14:58 smithi006 bash[26022]: cluster 2023-11-17T06:14:56.691790+0000 mgr.smithi006.odexej (mgr.14176) 1922 : cluster [DBG] pgmap v1305: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:00 smithi006 bash[26022]: cluster 2023-11-17T06:14:58.692958+0000 mgr.smithi006.odexej (mgr.14176) 1923 : cluster [DBG] pgmap v1306: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:02 smithi006 bash[26022]: cluster 2023-11-17T06:15:00.694396+0000 mgr.smithi006.odexej (mgr.14176) 1924 : cluster [DBG] pgmap v1307: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:14 smithi156 bash[29655]: cluster 2023-11-17T06:14:12.664691+0000 mgr.smithi006.odexej (mgr.14176) 1899 : cluster [DBG] pgmap v1283: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:14 smithi156 bash[29655]: audit 2023-11-17T06:14:13.135051+0000 mgr.smithi006.odexej (mgr.14176) 1900 : audit [DBG] from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:16 smithi156 bash[29655]: cluster 2023-11-17T06:14:14.666119+0000 mgr.smithi006.odexej (mgr.14176) 1901 : cluster [DBG] pgmap v1284: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:18 smithi156 bash[29655]: cluster 2023-11-17T06:14:16.667148+0000 mgr.smithi006.odexej (mgr.14176) 1902 : cluster [DBG] pgmap v1285: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:20 smithi156 bash[29655]: cluster 2023-11-17T06:14:18.668324+0000 mgr.smithi006.odexej (mgr.14176) 1903 : cluster [DBG] pgmap v1286: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.567 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:20 smithi156 bash[29655]: audit 2023-11-17T06:14:19.755134+0000 mon.smithi006 (mon.0) 1139 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:22 smithi156 bash[29655]: cluster 2023-11-17T06:14:20.669634+0000 mgr.smithi006.odexej (mgr.14176) 1904 : cluster [DBG] pgmap v1287: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:24 smithi156 bash[29655]: cluster 2023-11-17T06:14:22.670672+0000 mgr.smithi006.odexej (mgr.14176) 1905 : cluster [DBG] pgmap v1288: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:26 smithi156 bash[29655]: cluster 2023-11-17T06:14:24.672132+0000 mgr.smithi006.odexej (mgr.14176) 1906 : cluster [DBG] pgmap v1289: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:28 smithi156 bash[29655]: cluster 2023-11-17T06:14:26.673152+0000 mgr.smithi006.odexej (mgr.14176) 1907 : cluster [DBG] pgmap v1290: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:30 smithi156 bash[29655]: cluster 2023-11-17T06:14:28.674256+0000 mgr.smithi006.odexej (mgr.14176) 1908 : cluster [DBG] pgmap v1291: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:32 smithi156 bash[29655]: cluster 2023-11-17T06:14:30.675624+0000 mgr.smithi006.odexej (mgr.14176) 1909 : cluster [DBG] pgmap v1292: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:34 smithi156 bash[29655]: cluster 2023-11-17T06:14:32.676729+0000 mgr.smithi006.odexej (mgr.14176) 1910 : cluster [DBG] pgmap v1293: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:36 smithi156 bash[29655]: cluster 2023-11-17T06:14:34.678101+0000 mgr.smithi006.odexej (mgr.14176) 1911 : cluster [DBG] pgmap v1294: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:38 smithi156 bash[29655]: cluster 2023-11-17T06:14:36.679119+0000 mgr.smithi006.odexej (mgr.14176) 1912 : cluster [DBG] pgmap v1295: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:40 smithi156 bash[29655]: cluster 2023-11-17T06:14:38.680567+0000 mgr.smithi006.odexej (mgr.14176) 1913 : cluster [DBG] pgmap v1296: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:42 smithi156 bash[29655]: cluster 2023-11-17T06:14:40.682101+0000 mgr.smithi006.odexej (mgr.14176) 1914 : cluster [DBG] pgmap v1297: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.568 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:44 smithi156 bash[29655]: cluster 2023-11-17T06:14:42.683178+0000 mgr.smithi006.odexej (mgr.14176) 1915 : cluster [DBG] pgmap v1298: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.569 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:46 smithi156 bash[29655]: cluster 2023-11-17T06:14:44.684626+0000 mgr.smithi006.odexej (mgr.14176) 1916 : cluster [DBG] pgmap v1299: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.569 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:48 smithi156 bash[29655]: cluster 2023-11-17T06:14:46.685797+0000 mgr.smithi006.odexej (mgr.14176) 1917 : cluster [DBG] pgmap v1300: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.569 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:50 smithi156 bash[29655]: cluster 2023-11-17T06:14:48.686955+0000 mgr.smithi006.odexej (mgr.14176) 1918 : cluster [DBG] pgmap v1301: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.569 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:52 smithi156 bash[29655]: cluster 2023-11-17T06:14:50.688296+0000 mgr.smithi006.odexej (mgr.14176) 1919 : cluster [DBG] pgmap v1302: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.569 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:54 smithi156 bash[29655]: cluster 2023-11-17T06:14:52.689314+0000 mgr.smithi006.odexej (mgr.14176) 1920 : cluster [DBG] pgmap v1303: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:03.598 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:56 smithi156 bash[29655]: cluster 2023-11-17T06:14:54.690751+0000 mgr.smithi006.odexej (mgr.14176) 1921 : cluster [DBG] pgmap v1304: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:14:58 smithi156 bash[29655]: cluster 2023-11-17T06:14:56.691790+0000 mgr.smithi006.odexej (mgr.14176) 1922 : cluster [DBG] pgmap v1305: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:00 smithi156 bash[29655]: cluster 2023-11-17T06:14:58.692958+0000 mgr.smithi006.odexej (mgr.14176) 1923 : cluster [DBG] pgmap v1306: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:03.719 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:02 smithi156 bash[29655]: cluster 2023-11-17T06:15:00.694396+0000 mgr.smithi006.odexej (mgr.14176) 1924 : cluster [DBG] pgmap v1307: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:04.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:04 smithi006 bash[26022]: cluster 2023-11-17T06:15:02.695463+0000 mgr.smithi006.odexej (mgr.14176) 1925 : cluster [DBG] pgmap v1308: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:04.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:04 smithi156 bash[29655]: cluster 2023-11-17T06:15:02.695463+0000 mgr.smithi006.odexej (mgr.14176) 1925 : cluster [DBG] pgmap v1308: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:06.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:06 smithi006 bash[26022]: cluster 2023-11-17T06:15:04.696902+0000 mgr.smithi006.odexej (mgr.14176) 1926 : cluster [DBG] pgmap v1309: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:06.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:06 smithi156 bash[29655]: cluster 2023-11-17T06:15:04.696902+0000 mgr.smithi006.odexej (mgr.14176) 1926 : cluster [DBG] pgmap v1309: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:06.715 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:06.715 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:06.211705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\nDeploy daemon haproxy.nfs.foo.smithi156.oikrsx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.658669Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\nDeploy daemon haproxy.nfs.foo.smithi006.iaeoyq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.660594Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:07.323 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:08.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:08.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:08 smithi006 bash[26022]: cluster 2023-11-17T06:15:06.697887+0000 mgr.smithi006.odexej (mgr.14176) 1927 : cluster [DBG] pgmap v1310: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:08.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:08 smithi006 bash[26022]: audit 2023-11-17T06:15:06.699685+0000 mgr.smithi006.odexej (mgr.14176) 1928 : audit [DBG] from='client.15574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:08.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:08 smithi156 bash[29655]: cluster 2023-11-17T06:15:06.697887+0000 mgr.smithi006.odexej (mgr.14176) 1927 : cluster [DBG] pgmap v1310: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:08.746 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:08 smithi156 bash[29655]: audit 2023-11-17T06:15:06.699685+0000 mgr.smithi006.odexej (mgr.14176) 1928 : audit [DBG] from='client.15574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:10.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:10 smithi006 bash[26022]: cluster 2023-11-17T06:15:08.699073+0000 mgr.smithi006.odexej (mgr.14176) 1929 : cluster [DBG] pgmap v1311: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:10.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:10 smithi156 bash[29655]: cluster 2023-11-17T06:15:08.699073+0000 mgr.smithi006.odexej (mgr.14176) 1929 : cluster [DBG] pgmap v1311: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:10.666127+0000 mon.smithi006 (mon.0) 1140 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:10.996379+0000 mon.smithi006 (mon.0) 1141 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:10.998037+0000 mon.smithi006 (mon.0) 1142 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:11.005947+0000 mon.smithi006 (mon.0) 1143 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:11.015179+0000 mon.smithi006 (mon.0) 1144 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:15:11.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:11 smithi006 bash[26022]: audit 2023-11-17T06:15:11.029354+0000 mon.smithi006 (mon.0) 1145 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:11.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:11.479 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:13:01.811888Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fheqny on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:06.211705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\nDeploy daemon haproxy.nfs.foo.smithi156.oikrsx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.658669Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\nDeploy daemon haproxy.nfs.foo.smithi006.iaeoyq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.660594Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:10.666127+0000 mon.smithi006 (mon.0) 1140 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:10.996379+0000 mon.smithi006 (mon.0) 1141 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:10.998037+0000 mon.smithi006 (mon.0) 1142 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:11.005947+0000 mon.smithi006 (mon.0) 1143 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:11.015179+0000 mon.smithi006 (mon.0) 1144 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:15:11.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:11 smithi156 bash[29655]: audit 2023-11-17T06:15:11.029354+0000 mon.smithi006 (mon.0) 1145 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:12.095 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:12 smithi006 bash[26022]: cluster 2023-11-17T06:15:10.700077+0000 mgr.smithi006.odexej (mgr.14176) 1930 : cluster [DBG] pgmap v1312: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:12 smithi006 bash[26022]: cluster 2023-11-17T06:15:11.006737+0000 mgr.smithi006.odexej (mgr.14176) 1931 : cluster [DBG] pgmap v1313: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:12 smithi006 bash[26022]: cephadm 2023-11-17T06:15:11.033367+0000 mgr.smithi006.odexej (mgr.14176) 1932 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dofwcj on smithi156 2023-11-17T06:15:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:12 smithi006 bash[26022]: cluster 2023-11-17T06:15:11.240115+0000 mon.smithi006 (mon.0) 1146 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:15:12.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:12 smithi006 bash[26022]: cluster 2023-11-17T06:15:11.240168+0000 mon.smithi006 (mon.0) 1147 : cluster [INF] Cluster is now healthy 2023-11-17T06:15:12.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:12 smithi156 bash[29655]: cluster 2023-11-17T06:15:10.700077+0000 mgr.smithi006.odexej (mgr.14176) 1930 : cluster [DBG] pgmap v1312: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:12.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:12 smithi156 bash[29655]: cluster 2023-11-17T06:15:11.006737+0000 mgr.smithi006.odexej (mgr.14176) 1931 : cluster [DBG] pgmap v1313: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:12.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:12 smithi156 bash[29655]: cephadm 2023-11-17T06:15:11.033367+0000 mgr.smithi006.odexej (mgr.14176) 1932 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.dofwcj on smithi156 2023-11-17T06:15:12.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:12 smithi156 bash[29655]: cluster 2023-11-17T06:15:11.240115+0000 mon.smithi006 (mon.0) 1146 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:15:12.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:12 smithi156 bash[29655]: cluster 2023-11-17T06:15:11.240168+0000 mon.smithi006 (mon.0) 1147 : cluster [INF] Cluster is now healthy 2023-11-17T06:15:13.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:13.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:13 smithi006 bash[26022]: audit 2023-11-17T06:15:11.460612+0000 mgr.smithi006.odexej (mgr.14176) 1933 : audit [DBG] from='client.15578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:13 smithi156 bash[29655]: audit 2023-11-17T06:15:11.460612+0000 mgr.smithi006.odexej (mgr.14176) 1933 : audit [DBG] from='client.15578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:14.498 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:14 smithi006 bash[26022]: cluster 2023-11-17T06:15:13.007485+0000 mgr.smithi006.odexej (mgr.14176) 1934 : cluster [DBG] pgmap v1314: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:14.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:14 smithi156 bash[29655]: cluster 2023-11-17T06:15:13.007485+0000 mgr.smithi006.odexej (mgr.14176) 1934 : cluster [DBG] pgmap v1314: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:15.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:15 smithi156 bash[29655]: audit 2023-11-17T06:15:14.772508+0000 mon.smithi006 (mon.0) 1148 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dofwcj"}]: dispatch 2023-11-17T06:15:15.690 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:15 smithi006 bash[26022]: audit 2023-11-17T06:15:14.772508+0000 mon.smithi006 (mon.0) 1148 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.dofwcj"}]: dispatch 2023-11-17T06:15:16.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:16.500 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:06.211705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.oikrsx on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-oikrsx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.oikrsx\nDeploy daemon haproxy.nfs.foo.smithi156.oikrsx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.658669Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.iaeoyq on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-iaeoyq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.iaeoyq\nDeploy daemon haproxy.nfs.foo.smithi006.iaeoyq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:14:10.660594Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.nletru on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: cephadm 2023-11-17T06:15:14.771183+0000 mgr.smithi006.odexej (mgr.14176) 1935 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.563 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.564 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: cephadm 2023-11-17T06:15:14.771763+0000 mgr.smithi006.odexej (mgr.14176) 1936 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: cephadm 2023-11-17T06:15:14.774359+0000 mgr.smithi006.odexej (mgr.14176) 1937 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.565 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: cephadm 2023-11-17T06:15:14.779604+0000 mgr.smithi006.odexej (mgr.14176) 1938 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.alssai on smithi006 2023-11-17T06:15:16.566 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:16 smithi006 bash[26022]: cluster 2023-11-17T06:15:15.008323+0000 mgr.smithi006.odexej (mgr.14176) 1939 : cluster [DBG] pgmap v1315: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:16.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: cephadm 2023-11-17T06:15:14.771183+0000 mgr.smithi006.odexej (mgr.14176) 1935 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:15:16.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: cephadm 2023-11-17T06:15:14.771763+0000 mgr.smithi006.odexej (mgr.14176) 1936 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: cephadm 2023-11-17T06:15:14.774359+0000 mgr.smithi006.odexej (mgr.14176) 1937 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.dofwcj ... 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:16.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:16.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:16.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: cephadm 2023-11-17T06:15:14.779604+0000 mgr.smithi006.odexej (mgr.14176) 1938 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.alssai on smithi006 2023-11-17T06:15:16.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:16 smithi156 bash[29655]: cluster 2023-11-17T06:15:15.008323+0000 mgr.smithi006.odexej (mgr.14176) 1939 : cluster [DBG] pgmap v1315: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:17.131 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:18.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:18.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:18 smithi006 bash[26022]: audit 2023-11-17T06:15:16.484267+0000 mgr.smithi006.odexej (mgr.14176) 1940 : audit [DBG] from='client.15582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:18.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:18 smithi006 bash[26022]: cluster 2023-11-17T06:15:17.009072+0000 mgr.smithi006.odexej (mgr.14176) 1941 : cluster [DBG] pgmap v1316: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:18 smithi156 bash[29655]: audit 2023-11-17T06:15:16.484267+0000 mgr.smithi006.odexej (mgr.14176) 1940 : audit [DBG] from='client.15582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:18 smithi156 bash[29655]: cluster 2023-11-17T06:15:17.009072+0000 mgr.smithi006.odexej (mgr.14176) 1941 : cluster [DBG] pgmap v1316: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:19.546 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:19 smithi006 bash[26022]: audit 2023-11-17T06:15:18.984812+0000 mon.smithi006 (mon.0) 1149 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.alssai"}]: dispatch 2023-11-17T06:15:19.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:19 smithi156 bash[29655]: audit 2023-11-17T06:15:18.984812+0000 mon.smithi006 (mon.0) 1149 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.alssai"}]: dispatch 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cephadm 2023-11-17T06:15:18.983933+0000 mgr.smithi006.odexej (mgr.14176) 1942 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cephadm 2023-11-17T06:15:18.984273+0000 mgr.smithi006.odexej (mgr.14176) 1943 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cephadm 2023-11-17T06:15:18.986068+0000 mgr.smithi006.odexej (mgr.14176) 1944 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cephadm 2023-11-17T06:15:18.989081+0000 mgr.smithi006.odexej (mgr.14176) 1945 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cephadm 2023-11-17T06:15:18.992112+0000 mgr.smithi006.odexej (mgr.14176) 1946 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cluster 2023-11-17T06:15:18.993137+0000 mgr.smithi006.odexej (mgr.14176) 1947 : cluster [DBG] pgmap v1317: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:20.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:20 smithi006 bash[26022]: cluster 2023-11-17T06:15:19.295140+0000 mon.smithi006 (mon.0) 1150 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cephadm 2023-11-17T06:15:18.983933+0000 mgr.smithi006.odexej (mgr.14176) 1942 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cephadm 2023-11-17T06:15:18.984273+0000 mgr.smithi006.odexej (mgr.14176) 1943 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cephadm 2023-11-17T06:15:18.986068+0000 mgr.smithi006.odexej (mgr.14176) 1944 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.alssai ... 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cephadm 2023-11-17T06:15:18.989081+0000 mgr.smithi006.odexej (mgr.14176) 1945 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:15:20.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cephadm 2023-11-17T06:15:18.992112+0000 mgr.smithi006.odexej (mgr.14176) 1946 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:15:20.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cluster 2023-11-17T06:15:18.993137+0000 mgr.smithi006.odexej (mgr.14176) 1947 : cluster [DBG] pgmap v1317: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:20.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:20 smithi156 bash[29655]: cluster 2023-11-17T06:15:19.295140+0000 mon.smithi006 (mon.0) 1150 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:15:21.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:21.361 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:21.956 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:22.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:22 smithi006 bash[26022]: cluster 2023-11-17T06:15:20.994340+0000 mgr.smithi006.odexej (mgr.14176) 1948 : cluster [DBG] pgmap v1318: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:22.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:22 smithi006 bash[26022]: audit 2023-11-17T06:15:21.352252+0000 mgr.smithi006.odexej (mgr.14176) 1949 : audit [DBG] from='client.15586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:22.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:22 smithi006 bash[26022]: audit 2023-11-17T06:15:21.657628+0000 mon.smithi006 (mon.0) 1151 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:22.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:22 smithi156 bash[29655]: cluster 2023-11-17T06:15:20.994340+0000 mgr.smithi006.odexej (mgr.14176) 1948 : cluster [DBG] pgmap v1318: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:15:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:22 smithi156 bash[29655]: audit 2023-11-17T06:15:21.352252+0000 mgr.smithi006.odexej (mgr.14176) 1949 : audit [DBG] from='client.15586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:22.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:22 smithi156 bash[29655]: audit 2023-11-17T06:15:21.657628+0000 mon.smithi006 (mon.0) 1151 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:15:22.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:24.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:24 smithi006 bash[26022]: cluster 2023-11-17T06:15:22.995392+0000 mgr.smithi006.odexej (mgr.14176) 1950 : cluster [DBG] pgmap v1319: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:24.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:24 smithi156 bash[29655]: cluster 2023-11-17T06:15:22.995392+0000 mgr.smithi006.odexej (mgr.14176) 1950 : cluster [DBG] pgmap v1319: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:26.010 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:26.011 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:26.604 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:26 smithi006 bash[26022]: cluster 2023-11-17T06:15:24.996913+0000 mgr.smithi006.odexej (mgr.14176) 1951 : cluster [DBG] pgmap v1320: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:26.605 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:26.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:26 smithi156 bash[29655]: cluster 2023-11-17T06:15:24.996913+0000 mgr.smithi006.odexej (mgr.14176) 1951 : cluster [DBG] pgmap v1320: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:27.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:27.619 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:27 smithi006 bash[26022]: audit 2023-11-17T06:15:26.001638+0000 mgr.smithi006.odexej (mgr.14176) 1952 : audit [DBG] from='client.15590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:27.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:27 smithi156 bash[29655]: audit 2023-11-17T06:15:26.001638+0000 mgr.smithi006.odexej (mgr.14176) 1952 : audit [DBG] from='client.15590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:28.646 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:28 smithi006 bash[26022]: cluster 2023-11-17T06:15:26.997945+0000 mgr.smithi006.odexej (mgr.14176) 1953 : cluster [DBG] pgmap v1321: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:28.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:28 smithi156 bash[29655]: cluster 2023-11-17T06:15:26.997945+0000 mgr.smithi006.odexej (mgr.14176) 1953 : cluster [DBG] pgmap v1321: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:30.648 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:30.649 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:30.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:30 smithi006 bash[26022]: cluster 2023-11-17T06:15:28.999161+0000 mgr.smithi006.odexej (mgr.14176) 1954 : cluster [DBG] pgmap v1322: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:30.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:30 smithi156 bash[29655]: cluster 2023-11-17T06:15:28.999161+0000 mgr.smithi006.odexej (mgr.14176) 1954 : cluster [DBG] pgmap v1322: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:31.251 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:32.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:32.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:32 smithi006 bash[26022]: audit 2023-11-17T06:15:30.633707+0000 mgr.smithi006.odexej (mgr.14176) 1955 : audit [DBG] from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:32 smithi006 bash[26022]: cluster 2023-11-17T06:15:31.000578+0000 mgr.smithi006.odexej (mgr.14176) 1956 : cluster [DBG] pgmap v1323: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:32 smithi156 bash[29655]: audit 2023-11-17T06:15:30.633707+0000 mgr.smithi006.odexej (mgr.14176) 1955 : audit [DBG] from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:32.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:32 smithi156 bash[29655]: cluster 2023-11-17T06:15:31.000578+0000 mgr.smithi006.odexej (mgr.14176) 1956 : cluster [DBG] pgmap v1323: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:34.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:34 smithi006 bash[26022]: cluster 2023-11-17T06:15:33.001683+0000 mgr.smithi006.odexej (mgr.14176) 1957 : cluster [DBG] pgmap v1324: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:34.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:34 smithi156 bash[29655]: cluster 2023-11-17T06:15:33.001683+0000 mgr.smithi006.odexej (mgr.14176) 1957 : cluster [DBG] pgmap v1324: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:35.384 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:35.986 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:36.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:36 smithi006 bash[26022]: cluster 2023-11-17T06:15:35.003165+0000 mgr.smithi006.odexej (mgr.14176) 1958 : cluster [DBG] pgmap v1325: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:36 smithi156 bash[29655]: cluster 2023-11-17T06:15:35.003165+0000 mgr.smithi006.odexej (mgr.14176) 1958 : cluster [DBG] pgmap v1325: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:36.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:37.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:37 smithi006 bash[26022]: audit 2023-11-17T06:15:35.372954+0000 mgr.smithi006.odexej (mgr.14176) 1959 : audit [DBG] from='client.15598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:37.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:37 smithi156 bash[29655]: audit 2023-11-17T06:15:35.372954+0000 mgr.smithi006.odexej (mgr.14176) 1959 : audit [DBG] from='client.15598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:38.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:38 smithi006 bash[26022]: cluster 2023-11-17T06:15:37.004565+0000 mgr.smithi006.odexej (mgr.14176) 1960 : cluster [DBG] pgmap v1326: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:38.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:38 smithi156 bash[29655]: cluster 2023-11-17T06:15:37.004565+0000 mgr.smithi006.odexej (mgr.14176) 1960 : cluster [DBG] pgmap v1326: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:39.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:39 smithi006 bash[26022]: cluster 2023-11-17T06:15:39.005614+0000 mgr.smithi006.odexej (mgr.14176) 1961 : cluster [DBG] pgmap v1327: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:39.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:39 smithi156 bash[29655]: cluster 2023-11-17T06:15:39.005614+0000 mgr.smithi006.odexej (mgr.14176) 1961 : cluster [DBG] pgmap v1327: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:40.171 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:40.171 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:40.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:40 smithi006 bash[26022]: audit 2023-11-17T06:15:40.154316+0000 mgr.smithi006.odexej (mgr.14176) 1962 : audit [DBG] from='client.15602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:40 smithi156 bash[29655]: audit 2023-11-17T06:15:40.154316+0000 mgr.smithi006.odexej (mgr.14176) 1962 : audit [DBG] from='client.15602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:40.722 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:41.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:41 smithi006 bash[26022]: cluster 2023-11-17T06:15:41.006979+0000 mgr.smithi006.odexej (mgr.14176) 1963 : cluster [DBG] pgmap v1328: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:41.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:41 smithi156 bash[29655]: cluster 2023-11-17T06:15:41.006979+0000 mgr.smithi006.odexej (mgr.14176) 1963 : cluster [DBG] pgmap v1328: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:41.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:44.332 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:44 smithi006 bash[26022]: cluster 2023-11-17T06:15:43.008060+0000 mgr.smithi006.odexej (mgr.14176) 1964 : cluster [DBG] pgmap v1329: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:44.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:44 smithi156 bash[29655]: cluster 2023-11-17T06:15:43.008060+0000 mgr.smithi006.odexej (mgr.14176) 1964 : cluster [DBG] pgmap v1329: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:44.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:44.763 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:45.387 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:46.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:46.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:46 smithi006 bash[26022]: audit 2023-11-17T06:15:44.747773+0000 mgr.smithi006.odexej (mgr.14176) 1965 : audit [DBG] from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:46.414 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:46 smithi006 bash[26022]: cluster 2023-11-17T06:15:45.009536+0000 mgr.smithi006.odexej (mgr.14176) 1966 : cluster [DBG] pgmap v1330: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:46.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:46 smithi156 bash[29655]: audit 2023-11-17T06:15:44.747773+0000 mgr.smithi006.odexej (mgr.14176) 1965 : audit [DBG] from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:46.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:46 smithi156 bash[29655]: cluster 2023-11-17T06:15:45.009536+0000 mgr.smithi006.odexej (mgr.14176) 1966 : cluster [DBG] pgmap v1330: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:48.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:48 smithi006 bash[26022]: cluster 2023-11-17T06:15:47.010686+0000 mgr.smithi006.odexej (mgr.14176) 1967 : cluster [DBG] pgmap v1331: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:48.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:48 smithi156 bash[29655]: cluster 2023-11-17T06:15:47.010686+0000 mgr.smithi006.odexej (mgr.14176) 1967 : cluster [DBG] pgmap v1331: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:49.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:49.388 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:49.954 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:50.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:50 smithi006 bash[26022]: cluster 2023-11-17T06:15:49.011790+0000 mgr.smithi006.odexej (mgr.14176) 1968 : cluster [DBG] pgmap v1332: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:50.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:50 smithi156 bash[29655]: cluster 2023-11-17T06:15:49.011790+0000 mgr.smithi006.odexej (mgr.14176) 1968 : cluster [DBG] pgmap v1332: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:50.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:51.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:51 smithi006 bash[26022]: audit 2023-11-17T06:15:49.374752+0000 mgr.smithi006.odexej (mgr.14176) 1969 : audit [DBG] from='client.15610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:51.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:51 smithi156 bash[29655]: audit 2023-11-17T06:15:49.374752+0000 mgr.smithi006.odexej (mgr.14176) 1969 : audit [DBG] from='client.15610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:52.320 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:52 smithi006 bash[26022]: cluster 2023-11-17T06:15:51.013225+0000 mgr.smithi006.odexej (mgr.14176) 1970 : cluster [DBG] pgmap v1333: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:52.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:52 smithi156 bash[29655]: cluster 2023-11-17T06:15:51.013225+0000 mgr.smithi006.odexej (mgr.14176) 1970 : cluster [DBG] pgmap v1333: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:54.021 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:54.021 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:54.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:54 smithi006 bash[26022]: cluster 2023-11-17T06:15:53.014295+0000 mgr.smithi006.odexej (mgr.14176) 1971 : cluster [DBG] pgmap v1334: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:54.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:54 smithi156 bash[29655]: cluster 2023-11-17T06:15:53.014295+0000 mgr.smithi006.odexej (mgr.14176) 1971 : cluster [DBG] pgmap v1334: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:15:54.642 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:15:55.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:55 smithi006 bash[26022]: audit 2023-11-17T06:15:54.004620+0000 mgr.smithi006.odexej (mgr.14176) 1972 : audit [DBG] from='client.15614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:55.439 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:55 smithi156 bash[29655]: audit 2023-11-17T06:15:54.004620+0000 mgr.smithi006.odexej (mgr.14176) 1972 : audit [DBG] from='client.15614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:15:55.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:15:56.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:56 smithi006 bash[26022]: cluster 2023-11-17T06:15:55.015806+0000 mgr.smithi006.odexej (mgr.14176) 1973 : cluster [DBG] pgmap v1335: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:56.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:56 smithi156 bash[29655]: cluster 2023-11-17T06:15:55.015806+0000 mgr.smithi006.odexej (mgr.14176) 1973 : cluster [DBG] pgmap v1335: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:58.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:15:58 smithi006 bash[26022]: cluster 2023-11-17T06:15:57.016980+0000 mgr.smithi006.odexej (mgr.14176) 1974 : cluster [DBG] pgmap v1336: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:58.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:15:58 smithi156 bash[29655]: cluster 2023-11-17T06:15:57.016980+0000 mgr.smithi006.odexej (mgr.14176) 1974 : cluster [DBG] pgmap v1336: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:15:58.708 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:15:58.708 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:15:59.303 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:00.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:00.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:00 smithi006 bash[26022]: audit 2023-11-17T06:15:58.692292+0000 mgr.smithi006.odexej (mgr.14176) 1975 : audit [DBG] from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:00.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:00 smithi006 bash[26022]: cluster 2023-11-17T06:15:59.018229+0000 mgr.smithi006.odexej (mgr.14176) 1976 : cluster [DBG] pgmap v1337: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:00.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:00 smithi156 bash[29655]: audit 2023-11-17T06:15:58.692292+0000 mgr.smithi006.odexej (mgr.14176) 1975 : audit [DBG] from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:00.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:00 smithi156 bash[29655]: cluster 2023-11-17T06:15:59.018229+0000 mgr.smithi006.odexej (mgr.14176) 1976 : cluster [DBG] pgmap v1337: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:02.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:02 smithi006 bash[26022]: cluster 2023-11-17T06:16:01.019613+0000 mgr.smithi006.odexej (mgr.14176) 1977 : cluster [DBG] pgmap v1338: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:02.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:02 smithi156 bash[29655]: cluster 2023-11-17T06:16:01.019613+0000 mgr.smithi006.odexej (mgr.14176) 1977 : cluster [DBG] pgmap v1338: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:03.451 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:03.451 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:04.048 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:04.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:04 smithi006 bash[26022]: cluster 2023-11-17T06:16:03.020595+0000 mgr.smithi006.odexej (mgr.14176) 1978 : cluster [DBG] pgmap v1339: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:04.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:04 smithi156 bash[29655]: cluster 2023-11-17T06:16:03.020595+0000 mgr.smithi006.odexej (mgr.14176) 1978 : cluster [DBG] pgmap v1339: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:05.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:05.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:05 smithi006 bash[26022]: audit 2023-11-17T06:16:03.434893+0000 mgr.smithi006.odexej (mgr.14176) 1979 : audit [DBG] from='client.15622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:05.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:05 smithi156 bash[29655]: audit 2023-11-17T06:16:03.434893+0000 mgr.smithi006.odexej (mgr.14176) 1979 : audit [DBG] from='client.15622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:06.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:06 smithi006 bash[26022]: cluster 2023-11-17T06:16:05.021979+0000 mgr.smithi006.odexej (mgr.14176) 1980 : cluster [DBG] pgmap v1340: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:06.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:06 smithi156 bash[29655]: cluster 2023-11-17T06:16:05.021979+0000 mgr.smithi006.odexej (mgr.14176) 1980 : cluster [DBG] pgmap v1340: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:08.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:08.043 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:08.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:08 smithi006 bash[26022]: cluster 2023-11-17T06:16:07.023051+0000 mgr.smithi006.odexej (mgr.14176) 1981 : cluster [DBG] pgmap v1341: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:08.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:08 smithi156 bash[29655]: cluster 2023-11-17T06:16:07.023051+0000 mgr.smithi006.odexej (mgr.14176) 1981 : cluster [DBG] pgmap v1341: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:08.682 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:09.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:09 smithi006 bash[26022]: audit 2023-11-17T06:16:08.027261+0000 mgr.smithi006.odexej (mgr.14176) 1982 : audit [DBG] from='client.15626 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:09.434 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:09 smithi156 bash[29655]: audit 2023-11-17T06:16:08.027261+0000 mgr.smithi006.odexej (mgr.14176) 1982 : audit [DBG] from='client.15626 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:09.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:10.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:10 smithi006 bash[26022]: cluster 2023-11-17T06:16:09.024346+0000 mgr.smithi006.odexej (mgr.14176) 1983 : cluster [DBG] pgmap v1342: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:10.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:10 smithi156 bash[29655]: cluster 2023-11-17T06:16:09.024346+0000 mgr.smithi006.odexej (mgr.14176) 1983 : cluster [DBG] pgmap v1342: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:12.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:12 smithi006 bash[26022]: cluster 2023-11-17T06:16:11.025772+0000 mgr.smithi006.odexej (mgr.14176) 1984 : cluster [DBG] pgmap v1343: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:12.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:12 smithi156 bash[29655]: cluster 2023-11-17T06:16:11.025772+0000 mgr.smithi006.odexej (mgr.14176) 1984 : cluster [DBG] pgmap v1343: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:12.830 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:12.830 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:13.476 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:14.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:14 smithi156 bash[29655]: audit 2023-11-17T06:16:12.821880+0000 mgr.smithi006.odexej (mgr.14176) 1985 : audit [DBG] from='client.15630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:14.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:14 smithi156 bash[29655]: cluster 2023-11-17T06:16:13.026432+0000 mgr.smithi006.odexej (mgr.14176) 1986 : cluster [DBG] pgmap v1344: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:14.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:14.490 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:14 smithi006 bash[26022]: audit 2023-11-17T06:16:12.821880+0000 mgr.smithi006.odexej (mgr.14176) 1985 : audit [DBG] from='client.15630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:14.491 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:14 smithi006 bash[26022]: cluster 2023-11-17T06:16:13.026432+0000 mgr.smithi006.odexej (mgr.14176) 1986 : cluster [DBG] pgmap v1344: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:16.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:16 smithi156 bash[29655]: cluster 2023-11-17T06:16:15.027644+0000 mgr.smithi006.odexej (mgr.14176) 1987 : cluster [DBG] pgmap v1345: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:16.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:16 smithi006 bash[26022]: cluster 2023-11-17T06:16:15.027644+0000 mgr.smithi006.odexej (mgr.14176) 1987 : cluster [DBG] pgmap v1345: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:17.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:17.631 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:18.238 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:18.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:18 smithi006 bash[26022]: cluster 2023-11-17T06:16:17.028565+0000 mgr.smithi006.odexej (mgr.14176) 1988 : cluster [DBG] pgmap v1346: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:18.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:18 smithi156 bash[29655]: cluster 2023-11-17T06:16:17.028565+0000 mgr.smithi006.odexej (mgr.14176) 1988 : cluster [DBG] pgmap v1346: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:19.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:19.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:19 smithi006 bash[26022]: audit 2023-11-17T06:16:17.622679+0000 mgr.smithi006.odexej (mgr.14176) 1989 : audit [DBG] from='client.15634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:19.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:19 smithi006 bash[26022]: audit 2023-11-17T06:16:18.996206+0000 mon.smithi006 (mon.0) 1152 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:16:19.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:19 smithi156 bash[29655]: audit 2023-11-17T06:16:17.622679+0000 mgr.smithi006.odexej (mgr.14176) 1989 : audit [DBG] from='client.15634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:19.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:19 smithi156 bash[29655]: audit 2023-11-17T06:16:18.996206+0000 mon.smithi006 (mon.0) 1152 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:16:20.577 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: cluster 2023-11-17T06:16:19.029859+0000 mgr.smithi006.odexej (mgr.14176) 1990 : cluster [DBG] pgmap v1347: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.300109+0000 mon.smithi006 (mon.0) 1153 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.617881+0000 mon.smithi006 (mon.0) 1154 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.619949+0000 mon.smithi006 (mon.0) 1155 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.630368+0000 mon.smithi006 (mon.0) 1156 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.639667+0000 mon.smithi006 (mon.0) 1157 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:16:20.578 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:20 smithi006 bash[26022]: audit 2023-11-17T06:16:19.652515+0000 mon.smithi006 (mon.0) 1158 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: cluster 2023-11-17T06:16:19.029859+0000 mgr.smithi006.odexej (mgr.14176) 1990 : cluster [DBG] pgmap v1347: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.300109+0000 mon.smithi006 (mon.0) 1153 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.617881+0000 mon.smithi006 (mon.0) 1154 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.619949+0000 mon.smithi006 (mon.0) 1155 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.630368+0000 mon.smithi006 (mon.0) 1156 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.639667+0000 mon.smithi006 (mon.0) 1157 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:16:20.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:20 smithi156 bash[29655]: audit 2023-11-17T06:16:19.652515+0000 mon.smithi006 (mon.0) 1158 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:21.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:21 smithi006 bash[26022]: cluster 2023-11-17T06:16:19.631926+0000 mgr.smithi006.odexej (mgr.14176) 1991 : cluster [DBG] pgmap v1348: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:21.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:21 smithi006 bash[26022]: cephadm 2023-11-17T06:16:19.656032+0000 mgr.smithi006.odexej (mgr.14176) 1992 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.wysvwj on smithi156 2023-11-17T06:16:21.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:21 smithi006 bash[26022]: cluster 2023-11-17T06:16:20.626518+0000 mon.smithi006 (mon.0) 1159 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:16:21.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:21 smithi006 bash[26022]: cluster 2023-11-17T06:16:20.626590+0000 mon.smithi006 (mon.0) 1160 : cluster [INF] Cluster is now healthy 2023-11-17T06:16:21.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:21 smithi156 bash[29655]: cluster 2023-11-17T06:16:19.631926+0000 mgr.smithi006.odexej (mgr.14176) 1991 : cluster [DBG] pgmap v1348: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:21.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:21 smithi156 bash[29655]: cephadm 2023-11-17T06:16:19.656032+0000 mgr.smithi006.odexej (mgr.14176) 1992 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.wysvwj on smithi156 2023-11-17T06:16:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:21 smithi156 bash[29655]: cluster 2023-11-17T06:16:20.626518+0000 mon.smithi006 (mon.0) 1159 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:16:21.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:21 smithi156 bash[29655]: cluster 2023-11-17T06:16:20.626590+0000 mon.smithi006 (mon.0) 1160 : cluster [INF] Cluster is now healthy 2023-11-17T06:16:22.340 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:22.340 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:14:10.662226Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wkcfoa on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:14.774020Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.dofwcj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-dofwcj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.dofwcj\nDeploy daemon haproxy.nfs.foo.smithi156.dofwcj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.985900Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.alssai on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-alssai\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.alssai\nDeploy daemon haproxy.nfs.foo.smithi006.alssai ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:15:18.988910Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.fanrpo on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:22.943 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:23.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:23 smithi156 bash[29655]: cluster 2023-11-17T06:16:21.632742+0000 mgr.smithi006.odexej (mgr.14176) 1993 : cluster [DBG] pgmap v1349: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:23.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:23 smithi156 bash[29655]: audit 2023-11-17T06:16:22.325239+0000 mgr.smithi006.odexej (mgr.14176) 1994 : audit [DBG] from='client.15638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:23.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:23 smithi006 bash[26022]: cluster 2023-11-17T06:16:21.632742+0000 mgr.smithi006.odexej (mgr.14176) 1993 : cluster [DBG] pgmap v1349: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:23.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:23 smithi006 bash[26022]: audit 2023-11-17T06:16:22.325239+0000 mgr.smithi006.odexej (mgr.14176) 1994 : audit [DBG] from='client.15638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:23.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:24.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: cephadm 2023-11-17T06:16:23.374724+0000 mgr.smithi006.odexej (mgr.14176) 1995 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:16:24.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: cephadm 2023-11-17T06:16:23.375370+0000 mgr.smithi006.odexej (mgr.14176) 1996 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: audit 2023-11-17T06:16:23.376104+0000 mon.smithi006 (mon.0) 1161 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.wysvwj"}]: dispatch 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: cephadm 2023-11-17T06:16:23.378060+0000 mgr.smithi006.odexej (mgr.14176) 1997 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:24 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: cephadm 2023-11-17T06:16:23.374724+0000 mgr.smithi006.odexej (mgr.14176) 1995 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: cephadm 2023-11-17T06:16:23.375370+0000 mgr.smithi006.odexej (mgr.14176) 1996 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: audit 2023-11-17T06:16:23.376104+0000 mon.smithi006 (mon.0) 1161 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.wysvwj"}]: dispatch 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: cephadm 2023-11-17T06:16:23.378060+0000 mgr.smithi006.odexej (mgr.14176) 1997 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.wysvwj ... 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:24.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:24 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:25.553 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:25 smithi006 bash[26022]: cephadm 2023-11-17T06:16:23.383213+0000 mgr.smithi006.odexej (mgr.14176) 1998 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.hlkkjm on smithi006 2023-11-17T06:16:25.553 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:25 smithi006 bash[26022]: cluster 2023-11-17T06:16:23.633390+0000 mgr.smithi006.odexej (mgr.14176) 1999 : cluster [DBG] pgmap v1350: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:25.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:25 smithi156 bash[29655]: cephadm 2023-11-17T06:16:23.383213+0000 mgr.smithi006.odexej (mgr.14176) 1998 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.hlkkjm on smithi006 2023-11-17T06:16:25.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:25 smithi156 bash[29655]: cluster 2023-11-17T06:16:23.633390+0000 mgr.smithi006.odexej (mgr.14176) 1999 : cluster [DBG] pgmap v1350: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:27.239 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:27.240 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:27.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:27 smithi006 bash[26022]: cluster 2023-11-17T06:16:25.634476+0000 mgr.smithi006.odexej (mgr.14176) 2000 : cluster [DBG] pgmap v1351: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:27.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:27 smithi006 bash[26022]: audit 2023-11-17T06:16:27.132525+0000 mon.smithi006 (mon.0) 1162 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.hlkkjm"}]: dispatch 2023-11-17T06:16:27.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:27 smithi156 bash[29655]: cluster 2023-11-17T06:16:25.634476+0000 mgr.smithi006.odexej (mgr.14176) 2000 : cluster [DBG] pgmap v1351: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2023-11-17T06:16:27.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:27 smithi156 bash[29655]: audit 2023-11-17T06:16:27.132525+0000 mon.smithi006 (mon.0) 1162 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.hlkkjm"}]: dispatch 2023-11-17T06:16:27.874 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cephadm 2023-11-17T06:16:27.131759+0000 mgr.smithi006.odexej (mgr.14176) 2001 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cephadm 2023-11-17T06:16:27.132044+0000 mgr.smithi006.odexej (mgr.14176) 2002 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cephadm 2023-11-17T06:16:27.133661+0000 mgr.smithi006.odexej (mgr.14176) 2003 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cephadm 2023-11-17T06:16:27.136092+0000 mgr.smithi006.odexej (mgr.14176) 2004 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cephadm 2023-11-17T06:16:27.138227+0000 mgr.smithi006.odexej (mgr.14176) 2005 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cluster 2023-11-17T06:16:27.139154+0000 mgr.smithi006.odexej (mgr.14176) 2006 : cluster [DBG] pgmap v1352: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: audit 2023-11-17T06:16:27.221181+0000 mgr.smithi006.odexej (mgr.14176) 2007 : audit [DBG] from='client.25369 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:28.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:28 smithi006 bash[26022]: cluster 2023-11-17T06:16:28.059142+0000 mon.smithi006 (mon.0) 1163 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cephadm 2023-11-17T06:16:27.131759+0000 mgr.smithi006.odexej (mgr.14176) 2001 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cephadm 2023-11-17T06:16:27.132044+0000 mgr.smithi006.odexej (mgr.14176) 2002 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cephadm 2023-11-17T06:16:27.133661+0000 mgr.smithi006.odexej (mgr.14176) 2003 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.hlkkjm ... 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:16:28.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cephadm 2023-11-17T06:16:27.136092+0000 mgr.smithi006.odexej (mgr.14176) 2004 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:16:28.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cephadm 2023-11-17T06:16:27.138227+0000 mgr.smithi006.odexej (mgr.14176) 2005 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:16:28.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cluster 2023-11-17T06:16:27.139154+0000 mgr.smithi006.odexej (mgr.14176) 2006 : cluster [DBG] pgmap v1352: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:16:28.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: audit 2023-11-17T06:16:27.221181+0000 mgr.smithi006.odexej (mgr.14176) 2007 : audit [DBG] from='client.25369 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:28.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:28 smithi156 bash[29655]: cluster 2023-11-17T06:16:28.059142+0000 mon.smithi006 (mon.0) 1163 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:16:28.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:30.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:30 smithi006 bash[26022]: cluster 2023-11-17T06:16:29.140321+0000 mgr.smithi006.odexej (mgr.14176) 2008 : cluster [DBG] pgmap v1353: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:16:30.707 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:30 smithi156 bash[29655]: cluster 2023-11-17T06:16:29.140321+0000 mgr.smithi006.odexej (mgr.14176) 2008 : cluster [DBG] pgmap v1353: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 101 B/s rd, 0 op/s 2023-11-17T06:16:31.927 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:31.927 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:32.530 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:32.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:32 smithi006 bash[26022]: cluster 2023-11-17T06:16:31.141742+0000 mgr.smithi006.odexej (mgr.14176) 2009 : cluster [DBG] pgmap v1354: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:32.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:32 smithi006 bash[26022]: audit 2023-11-17T06:16:31.709795+0000 mon.smithi006 (mon.0) 1164 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:32 smithi156 bash[29655]: cluster 2023-11-17T06:16:31.141742+0000 mgr.smithi006.odexej (mgr.14176) 2009 : cluster [DBG] pgmap v1354: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:32.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:32 smithi156 bash[29655]: audit 2023-11-17T06:16:31.709795+0000 mon.smithi006 (mon.0) 1164 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:16:33.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:33.623 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:33 smithi156 bash[29655]: audit 2023-11-17T06:16:31.914002+0000 mgr.smithi006.odexej (mgr.14176) 2010 : audit [DBG] from='client.15646 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:33.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:33 smithi006 bash[26022]: audit 2023-11-17T06:16:31.914002+0000 mgr.smithi006.odexej (mgr.14176) 2010 : audit [DBG] from='client.15646 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:34.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:34 smithi006 bash[26022]: cluster 2023-11-17T06:16:33.142843+0000 mgr.smithi006.odexej (mgr.14176) 2011 : cluster [DBG] pgmap v1355: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:16:34.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:34 smithi156 bash[29655]: cluster 2023-11-17T06:16:33.142843+0000 mgr.smithi006.odexej (mgr.14176) 2011 : cluster [DBG] pgmap v1355: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:16:36.626 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:36.626 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:36.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:36 smithi006 bash[26022]: cluster 2023-11-17T06:16:35.144285+0000 mgr.smithi006.odexej (mgr.14176) 2012 : cluster [DBG] pgmap v1356: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:36.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:36 smithi156 bash[29655]: cluster 2023-11-17T06:16:35.144285+0000 mgr.smithi006.odexej (mgr.14176) 2012 : cluster [DBG] pgmap v1356: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:37.247 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:38.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:38.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:38 smithi006 bash[26022]: audit 2023-11-17T06:16:36.618297+0000 mgr.smithi006.odexej (mgr.14176) 2013 : audit [DBG] from='client.15650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:38.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:38 smithi006 bash[26022]: cluster 2023-11-17T06:16:37.145313+0000 mgr.smithi006.odexej (mgr.14176) 2014 : cluster [DBG] pgmap v1357: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:38.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:38 smithi156 bash[29655]: audit 2023-11-17T06:16:36.618297+0000 mgr.smithi006.odexej (mgr.14176) 2013 : audit [DBG] from='client.15650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:38.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:38 smithi156 bash[29655]: cluster 2023-11-17T06:16:37.145313+0000 mgr.smithi006.odexej (mgr.14176) 2014 : cluster [DBG] pgmap v1357: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:16:40.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:40 smithi006 bash[26022]: cluster 2023-11-17T06:16:39.146481+0000 mgr.smithi006.odexej (mgr.14176) 2015 : cluster [DBG] pgmap v1358: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:40 smithi156 bash[29655]: cluster 2023-11-17T06:16:39.146481+0000 mgr.smithi006.odexej (mgr.14176) 2015 : cluster [DBG] pgmap v1358: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:41.333 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:41.941 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:42.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:42 smithi006 bash[26022]: cluster 2023-11-17T06:16:41.147899+0000 mgr.smithi006.odexej (mgr.14176) 2016 : cluster [DBG] pgmap v1359: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:42.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:42 smithi006 bash[26022]: audit 2023-11-17T06:16:41.324644+0000 mgr.smithi006.odexej (mgr.14176) 2017 : audit [DBG] from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:42.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:42 smithi156 bash[29655]: cluster 2023-11-17T06:16:41.147899+0000 mgr.smithi006.odexej (mgr.14176) 2016 : cluster [DBG] pgmap v1359: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:42.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:42 smithi156 bash[29655]: audit 2023-11-17T06:16:41.324644+0000 mgr.smithi006.odexej (mgr.14176) 2017 : audit [DBG] from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:42.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:44.339 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:44 smithi006 bash[26022]: cluster 2023-11-17T06:16:43.148554+0000 mgr.smithi006.odexej (mgr.14176) 2018 : cluster [DBG] pgmap v1360: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:44.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:44 smithi156 bash[29655]: cluster 2023-11-17T06:16:43.148554+0000 mgr.smithi006.odexej (mgr.14176) 2018 : cluster [DBG] pgmap v1360: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:45.985 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:45.985 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:46.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:46 smithi006 bash[26022]: cluster 2023-11-17T06:16:45.149954+0000 mgr.smithi006.odexej (mgr.14176) 2019 : cluster [DBG] pgmap v1361: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:46.576 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:46.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:46 smithi156 bash[29655]: cluster 2023-11-17T06:16:45.149954+0000 mgr.smithi006.odexej (mgr.14176) 2019 : cluster [DBG] pgmap v1361: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:47.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:47.590 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:47 smithi006 bash[26022]: audit 2023-11-17T06:16:45.968839+0000 mgr.smithi006.odexej (mgr.14176) 2020 : audit [DBG] from='client.15658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:47.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:47 smithi156 bash[29655]: audit 2023-11-17T06:16:45.968839+0000 mgr.smithi006.odexej (mgr.14176) 2020 : audit [DBG] from='client.15658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:48.621 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:48 smithi006 bash[26022]: cluster 2023-11-17T06:16:47.150996+0000 mgr.smithi006.odexej (mgr.14176) 2021 : cluster [DBG] pgmap v1362: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:48.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:48 smithi156 bash[29655]: cluster 2023-11-17T06:16:47.150996+0000 mgr.smithi006.odexej (mgr.14176) 2021 : cluster [DBG] pgmap v1362: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:50.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:50 smithi006 bash[26022]: cluster 2023-11-17T06:16:49.152227+0000 mgr.smithi006.odexej (mgr.14176) 2022 : cluster [DBG] pgmap v1363: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:50.476 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:50.476 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:50.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:50 smithi156 bash[29655]: cluster 2023-11-17T06:16:49.152227+0000 mgr.smithi006.odexej (mgr.14176) 2022 : cluster [DBG] pgmap v1363: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:51.126 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:52.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:52.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:52 smithi006 bash[26022]: audit 2023-11-17T06:16:50.462721+0000 mgr.smithi006.odexej (mgr.14176) 2023 : audit [DBG] from='client.15662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:52.417 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:52 smithi006 bash[26022]: cluster 2023-11-17T06:16:51.153569+0000 mgr.smithi006.odexej (mgr.14176) 2024 : cluster [DBG] pgmap v1364: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:52.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:52 smithi156 bash[29655]: audit 2023-11-17T06:16:50.462721+0000 mgr.smithi006.odexej (mgr.14176) 2023 : audit [DBG] from='client.15662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:52.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:52 smithi156 bash[29655]: cluster 2023-11-17T06:16:51.153569+0000 mgr.smithi006.odexej (mgr.14176) 2024 : cluster [DBG] pgmap v1364: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:54.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:54 smithi006 bash[26022]: cluster 2023-11-17T06:16:53.154701+0000 mgr.smithi006.odexej (mgr.14176) 2025 : cluster [DBG] pgmap v1365: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:54.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:54 smithi156 bash[29655]: cluster 2023-11-17T06:16:53.154701+0000 mgr.smithi006.odexej (mgr.14176) 2025 : cluster [DBG] pgmap v1365: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:16:55.331 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:16:55.331 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:16:55.984 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:16:56.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:56 smithi006 bash[26022]: cluster 2023-11-17T06:16:55.155930+0000 mgr.smithi006.odexej (mgr.14176) 2026 : cluster [DBG] pgmap v1366: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:56.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:56 smithi006 bash[26022]: audit 2023-11-17T06:16:55.323185+0000 mgr.smithi006.odexej (mgr.14176) 2027 : audit [DBG] from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:56.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:56 smithi156 bash[29655]: cluster 2023-11-17T06:16:55.155930+0000 mgr.smithi006.odexej (mgr.14176) 2026 : cluster [DBG] pgmap v1366: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:56.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:56 smithi156 bash[29655]: audit 2023-11-17T06:16:55.323185+0000 mgr.smithi006.odexej (mgr.14176) 2027 : audit [DBG] from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:16:56.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:16:58.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:16:58 smithi006 bash[26022]: cluster 2023-11-17T06:16:57.156916+0000 mgr.smithi006.odexej (mgr.14176) 2028 : cluster [DBG] pgmap v1367: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:16:58.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:16:58 smithi156 bash[29655]: cluster 2023-11-17T06:16:57.156916+0000 mgr.smithi006.odexej (mgr.14176) 2028 : cluster [DBG] pgmap v1367: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:00.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:17:00.111 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:17:00.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:00 smithi006 bash[26022]: cluster 2023-11-17T06:16:59.158097+0000 mgr.smithi006.odexej (mgr.14176) 2029 : cluster [DBG] pgmap v1368: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:00.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:00 smithi156 bash[29655]: cluster 2023-11-17T06:16:59.158097+0000 mgr.smithi006.odexej (mgr.14176) 2029 : cluster [DBG] pgmap v1368: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:00.751 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:17:01.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:01 smithi006 bash[26022]: audit 2023-11-17T06:17:00.095331+0000 mgr.smithi006.odexej (mgr.14176) 2030 : audit [DBG] from='client.15670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:01.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:01 smithi156 bash[29655]: audit 2023-11-17T06:17:00.095331+0000 mgr.smithi006.odexej (mgr.14176) 2030 : audit [DBG] from='client.15670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:01.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:17:02.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:02 smithi006 bash[26022]: cluster 2023-11-17T06:17:01.159450+0000 mgr.smithi006.odexej (mgr.14176) 2031 : cluster [DBG] pgmap v1369: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:02.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:02 smithi156 bash[29655]: cluster 2023-11-17T06:17:01.159450+0000 mgr.smithi006.odexej (mgr.14176) 2031 : cluster [DBG] pgmap v1369: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:04.488 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:04 smithi006 bash[26022]: cluster 2023-11-17T06:17:03.160317+0000 mgr.smithi006.odexej (mgr.14176) 2032 : cluster [DBG] pgmap v1370: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.650 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:17:44.650 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:15:18.991968Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.lrtkho on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:23.377718Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.wysvwj on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-wysvwj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.wysvwj\nDeploy daemon haproxy.nfs.foo.smithi156.wysvwj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.133526Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.hlkkjm on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-hlkkjm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.hlkkjm\nDeploy daemon haproxy.nfs.foo.smithi006.hlkkjm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:16:27.135972Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ywcksf on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:17:44.651 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:17:44.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:04 smithi156 bash[29655]: cluster 2023-11-17T06:17:03.160317+0000 mgr.smithi006.odexej (mgr.14176) 2032 : cluster [DBG] pgmap v1370: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:06 smithi156 bash[29655]: audit 2023-11-17T06:17:04.927338+0000 mgr.smithi006.odexej (mgr.14176) 2033 : audit [DBG] from='client.15674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:44.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:06 smithi156 bash[29655]: cluster 2023-11-17T06:17:05.161796+0000 mgr.smithi006.odexej (mgr.14176) 2034 : cluster [DBG] pgmap v1371: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.688 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:08 smithi156 bash[29655]: cluster 2023-11-17T06:17:07.162782+0000 mgr.smithi006.odexej (mgr.14176) 2035 : cluster [DBG] pgmap v1372: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:10 smithi156 bash[29655]: cluster 2023-11-17T06:17:09.163983+0000 mgr.smithi006.odexej (mgr.14176) 2036 : cluster [DBG] pgmap v1373: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:12 smithi156 bash[29655]: cluster 2023-11-17T06:17:11.165329+0000 mgr.smithi006.odexej (mgr.14176) 2037 : cluster [DBG] pgmap v1374: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:14 smithi156 bash[29655]: cluster 2023-11-17T06:17:13.166359+0000 mgr.smithi006.odexej (mgr.14176) 2038 : cluster [DBG] pgmap v1375: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:16 smithi156 bash[29655]: cluster 2023-11-17T06:17:15.167824+0000 mgr.smithi006.odexej (mgr.14176) 2039 : cluster [DBG] pgmap v1376: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:18 smithi156 bash[29655]: cluster 2023-11-17T06:17:17.168850+0000 mgr.smithi006.odexej (mgr.14176) 2040 : cluster [DBG] pgmap v1377: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:20 smithi156 bash[29655]: cluster 2023-11-17T06:17:19.170023+0000 mgr.smithi006.odexej (mgr.14176) 2041 : cluster [DBG] pgmap v1378: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:22 smithi156 bash[29655]: cluster 2023-11-17T06:17:21.171376+0000 mgr.smithi006.odexej (mgr.14176) 2042 : cluster [DBG] pgmap v1379: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:24 smithi156 bash[29655]: cluster 2023-11-17T06:17:23.172382+0000 mgr.smithi006.odexej (mgr.14176) 2043 : cluster [DBG] pgmap v1380: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:26 smithi156 bash[29655]: cluster 2023-11-17T06:17:25.173754+0000 mgr.smithi006.odexej (mgr.14176) 2044 : cluster [DBG] pgmap v1381: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.142828+0000 mon.smithi006 (mon.0) 1165 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: cluster 2023-11-17T06:17:27.174553+0000 mgr.smithi006.odexej (mgr.14176) 2045 : cluster [DBG] pgmap v1382: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.450391+0000 mon.smithi006 (mon.0) 1166 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:17:44.689 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.451850+0000 mon.smithi006 (mon.0) 1167 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:17:44.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.459848+0000 mon.smithi006 (mon.0) 1168 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.469071+0000 mon.smithi006 (mon.0) 1169 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:17:44.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:28 smithi156 bash[29655]: audit 2023-11-17T06:17:27.485367+0000 mon.smithi006 (mon.0) 1170 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:29 smithi156 bash[29655]: cluster 2023-11-17T06:17:27.460717+0000 mgr.smithi006.odexej (mgr.14176) 2046 : cluster [DBG] pgmap v1383: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.690 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:29 smithi156 bash[29655]: cephadm 2023-11-17T06:17:27.488439+0000 mgr.smithi006.odexej (mgr.14176) 2047 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ldxsxc on smithi156 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:29 smithi156 bash[29655]: cluster 2023-11-17T06:17:28.078601+0000 mon.smithi006 (mon.0) 1171 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:29 smithi156 bash[29655]: cluster 2023-11-17T06:17:28.078669+0000 mon.smithi006 (mon.0) 1172 : cluster [INF] Cluster is now healthy 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:31 smithi156 bash[29655]: cluster 2023-11-17T06:17:29.461817+0000 mgr.smithi006.odexej (mgr.14176) 2048 : cluster [DBG] pgmap v1384: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: cephadm 2023-11-17T06:17:31.215133+0000 mgr.smithi006.odexej (mgr.14176) 2049 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.729 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.730 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: cephadm 2023-11-17T06:17:31.215626+0000 mgr.smithi006.odexej (mgr.14176) 2050 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: audit 2023-11-17T06:17:31.216235+0000 mon.smithi006 (mon.0) 1173 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ldxsxc"}]: dispatch 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: cephadm 2023-11-17T06:17:31.218062+0000 mgr.smithi006.odexej (mgr.14176) 2051 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.731 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:32 smithi156 bash[29655]: cephadm 2023-11-17T06:17:31.223366+0000 mgr.smithi006.odexej (mgr.14176) 2052 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.ctpzby on smithi006 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:33 smithi156 bash[29655]: cluster 2023-11-17T06:17:31.462552+0000 mgr.smithi006.odexej (mgr.14176) 2053 : cluster [DBG] pgmap v1385: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:35 smithi156 bash[29655]: cluster 2023-11-17T06:17:33.463450+0000 mgr.smithi006.odexej (mgr.14176) 2054 : cluster [DBG] pgmap v1386: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:35 smithi156 bash[29655]: audit 2023-11-17T06:17:34.760285+0000 mon.smithi006 (mon.0) 1174 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.732 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:35 smithi156 bash[29655]: audit 2023-11-17T06:17:34.987063+0000 mon.smithi006 (mon.0) 1175 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.ctpzby"}]: dispatch 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cephadm 2023-11-17T06:17:34.985847+0000 mgr.smithi006.odexej (mgr.14176) 2055 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.733 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.734 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cephadm 2023-11-17T06:17:34.986398+0000 mgr.smithi006.odexej (mgr.14176) 2056 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cephadm 2023-11-17T06:17:34.988680+0000 mgr.smithi006.odexej (mgr.14176) 2057 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:17:44.735 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cephadm 2023-11-17T06:17:34.993085+0000 mgr.smithi006.odexej (mgr.14176) 2058 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cephadm 2023-11-17T06:17:34.997148+0000 mgr.smithi006.odexej (mgr.14176) 2059 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cluster 2023-11-17T06:17:34.998683+0000 mgr.smithi006.odexej (mgr.14176) 2060 : cluster [DBG] pgmap v1387: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:36 smithi156 bash[29655]: cluster 2023-11-17T06:17:35.756112+0000 mon.smithi006 (mon.0) 1176 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:37 smithi156 bash[29655]: audit 2023-11-17T06:17:36.764039+0000 mon.smithi006 (mon.0) 1177 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.736 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:37 smithi156 bash[29655]: cluster 2023-11-17T06:17:36.999718+0000 mgr.smithi006.odexej (mgr.14176) 2061 : cluster [DBG] pgmap v1388: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:17:44.737 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:40 smithi156 bash[29655]: cluster 2023-11-17T06:17:39.000854+0000 mgr.smithi006.odexej (mgr.14176) 2062 : cluster [DBG] pgmap v1389: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:44.737 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:42 smithi156 bash[29655]: cluster 2023-11-17T06:17:41.002290+0000 mgr.smithi006.odexej (mgr.14176) 2063 : cluster [DBG] pgmap v1390: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:44.756 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:44 smithi156 bash[29655]: cluster 2023-11-17T06:17:43.003360+0000 mgr.smithi006.odexej (mgr.14176) 2064 : cluster [DBG] pgmap v1391: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:06 smithi006 bash[26022]: audit 2023-11-17T06:17:04.927338+0000 mgr.smithi006.odexej (mgr.14176) 2033 : audit [DBG] from='client.15674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:06 smithi006 bash[26022]: cluster 2023-11-17T06:17:05.161796+0000 mgr.smithi006.odexej (mgr.14176) 2034 : cluster [DBG] pgmap v1371: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:08 smithi006 bash[26022]: cluster 2023-11-17T06:17:07.162782+0000 mgr.smithi006.odexej (mgr.14176) 2035 : cluster [DBG] pgmap v1372: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:10 smithi006 bash[26022]: cluster 2023-11-17T06:17:09.163983+0000 mgr.smithi006.odexej (mgr.14176) 2036 : cluster [DBG] pgmap v1373: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:12 smithi006 bash[26022]: cluster 2023-11-17T06:17:11.165329+0000 mgr.smithi006.odexej (mgr.14176) 2037 : cluster [DBG] pgmap v1374: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:14 smithi006 bash[26022]: cluster 2023-11-17T06:17:13.166359+0000 mgr.smithi006.odexej (mgr.14176) 2038 : cluster [DBG] pgmap v1375: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.757 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:16 smithi006 bash[26022]: cluster 2023-11-17T06:17:15.167824+0000 mgr.smithi006.odexej (mgr.14176) 2039 : cluster [DBG] pgmap v1376: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:18 smithi006 bash[26022]: cluster 2023-11-17T06:17:17.168850+0000 mgr.smithi006.odexej (mgr.14176) 2040 : cluster [DBG] pgmap v1377: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:20 smithi006 bash[26022]: cluster 2023-11-17T06:17:19.170023+0000 mgr.smithi006.odexej (mgr.14176) 2041 : cluster [DBG] pgmap v1378: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:22 smithi006 bash[26022]: cluster 2023-11-17T06:17:21.171376+0000 mgr.smithi006.odexej (mgr.14176) 2042 : cluster [DBG] pgmap v1379: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:24 smithi006 bash[26022]: cluster 2023-11-17T06:17:23.172382+0000 mgr.smithi006.odexej (mgr.14176) 2043 : cluster [DBG] pgmap v1380: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:26 smithi006 bash[26022]: cluster 2023-11-17T06:17:25.173754+0000 mgr.smithi006.odexej (mgr.14176) 2044 : cluster [DBG] pgmap v1381: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.142828+0000 mon.smithi006 (mon.0) 1165 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: cluster 2023-11-17T06:17:27.174553+0000 mgr.smithi006.odexej (mgr.14176) 2045 : cluster [DBG] pgmap v1382: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.450391+0000 mon.smithi006 (mon.0) 1166 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.451850+0000 mon.smithi006 (mon.0) 1167 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.459848+0000 mon.smithi006 (mon.0) 1168 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.469071+0000 mon.smithi006 (mon.0) 1169 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:28 smithi006 bash[26022]: audit 2023-11-17T06:17:27.485367+0000 mon.smithi006 (mon.0) 1170 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.758 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:29 smithi006 bash[26022]: cluster 2023-11-17T06:17:27.460717+0000 mgr.smithi006.odexej (mgr.14176) 2046 : cluster [DBG] pgmap v1383: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:29 smithi006 bash[26022]: cephadm 2023-11-17T06:17:27.488439+0000 mgr.smithi006.odexej (mgr.14176) 2047 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.ldxsxc on smithi156 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:29 smithi006 bash[26022]: cluster 2023-11-17T06:17:28.078601+0000 mon.smithi006 (mon.0) 1171 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:29 smithi006 bash[26022]: cluster 2023-11-17T06:17:28.078669+0000 mon.smithi006 (mon.0) 1172 : cluster [INF] Cluster is now healthy 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:31 smithi006 bash[26022]: cluster 2023-11-17T06:17:29.461817+0000 mgr.smithi006.odexej (mgr.14176) 2048 : cluster [DBG] pgmap v1384: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: cephadm 2023-11-17T06:17:31.215133+0000 mgr.smithi006.odexej (mgr.14176) 2049 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.759 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.760 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: cephadm 2023-11-17T06:17:31.215626+0000 mgr.smithi006.odexej (mgr.14176) 2050 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: audit 2023-11-17T06:17:31.216235+0000 mon.smithi006 (mon.0) 1173 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.ldxsxc"}]: dispatch 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: cephadm 2023-11-17T06:17:31.218062+0000 mgr.smithi006.odexej (mgr.14176) 2051 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.761 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.ldxsxc ... 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:32 smithi006 bash[26022]: cephadm 2023-11-17T06:17:31.223366+0000 mgr.smithi006.odexej (mgr.14176) 2052 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.ctpzby on smithi006 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:33 smithi006 bash[26022]: cluster 2023-11-17T06:17:31.462552+0000 mgr.smithi006.odexej (mgr.14176) 2053 : cluster [DBG] pgmap v1385: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:35 smithi006 bash[26022]: cluster 2023-11-17T06:17:33.463450+0000 mgr.smithi006.odexej (mgr.14176) 2054 : cluster [DBG] pgmap v1386: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:35 smithi006 bash[26022]: audit 2023-11-17T06:17:34.760285+0000 mon.smithi006 (mon.0) 1174 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:35 smithi006 bash[26022]: audit 2023-11-17T06:17:34.987063+0000 mon.smithi006 (mon.0) 1175 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.ctpzby"}]: dispatch 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cephadm 2023-11-17T06:17:34.985847+0000 mgr.smithi006.odexej (mgr.14176) 2055 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.762 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:17:44.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cephadm 2023-11-17T06:17:34.986398+0000 mgr.smithi006.odexej (mgr.14176) 2056 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cephadm 2023-11-17T06:17:34.988680+0000 mgr.smithi006.odexej (mgr.14176) 2057 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.ctpzby ... 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:17:44.765 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:17:44.783 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cephadm 2023-11-17T06:17:34.993085+0000 mgr.smithi006.odexej (mgr.14176) 2058 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cephadm 2023-11-17T06:17:34.997148+0000 mgr.smithi006.odexej (mgr.14176) 2059 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cluster 2023-11-17T06:17:34.998683+0000 mgr.smithi006.odexej (mgr.14176) 2060 : cluster [DBG] pgmap v1387: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:36 smithi006 bash[26022]: cluster 2023-11-17T06:17:35.756112+0000 mon.smithi006 (mon.0) 1176 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:37 smithi006 bash[26022]: audit 2023-11-17T06:17:36.764039+0000 mon.smithi006 (mon.0) 1177 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:37 smithi006 bash[26022]: cluster 2023-11-17T06:17:36.999718+0000 mgr.smithi006.odexej (mgr.14176) 2061 : cluster [DBG] pgmap v1388: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:40 smithi006 bash[26022]: cluster 2023-11-17T06:17:39.000854+0000 mgr.smithi006.odexej (mgr.14176) 2062 : cluster [DBG] pgmap v1389: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:42 smithi006 bash[26022]: cluster 2023-11-17T06:17:41.002290+0000 mgr.smithi006.odexej (mgr.14176) 2063 : cluster [DBG] pgmap v1390: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:44.784 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:44 smithi006 bash[26022]: cluster 2023-11-17T06:17:43.003360+0000 mgr.smithi006.odexej (mgr.14176) 2064 : cluster [DBG] pgmap v1391: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:17:45.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:17:46.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:46 smithi006 bash[26022]: cluster 2023-11-17T06:17:45.004796+0000 mgr.smithi006.odexej (mgr.14176) 2065 : cluster [DBG] pgmap v1392: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:46.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:46 smithi156 bash[29655]: cluster 2023-11-17T06:17:45.004796+0000 mgr.smithi006.odexej (mgr.14176) 2065 : cluster [DBG] pgmap v1392: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 88 B/s rd, 0 op/s 2023-11-17T06:17:48.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:48 smithi006 bash[26022]: cluster 2023-11-17T06:17:47.005824+0000 mgr.smithi006.odexej (mgr.14176) 2066 : cluster [DBG] pgmap v1393: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:48.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:48 smithi156 bash[29655]: cluster 2023-11-17T06:17:47.005824+0000 mgr.smithi006.odexej (mgr.14176) 2066 : cluster [DBG] pgmap v1393: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:48.872 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:17:48.872 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:17:49.458 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:17:50.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:50 smithi006 bash[26022]: audit 2023-11-17T06:17:48.855982+0000 mgr.smithi006.odexej (mgr.14176) 2067 : audit [DBG] from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:50.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:50 smithi006 bash[26022]: cluster 2023-11-17T06:17:49.006987+0000 mgr.smithi006.odexej (mgr.14176) 2068 : cluster [DBG] pgmap v1394: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:50.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:50 smithi006 bash[26022]: audit 2023-11-17T06:17:49.760858+0000 mon.smithi006 (mon.0) 1178 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:50.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:50 smithi156 bash[29655]: audit 2023-11-17T06:17:48.855982+0000 mgr.smithi006.odexej (mgr.14176) 2067 : audit [DBG] from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:50.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:50 smithi156 bash[29655]: cluster 2023-11-17T06:17:49.006987+0000 mgr.smithi006.odexej (mgr.14176) 2068 : cluster [DBG] pgmap v1394: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:50.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:50 smithi156 bash[29655]: audit 2023-11-17T06:17:49.760858+0000 mon.smithi006 (mon.0) 1178 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:17:50.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:17:52.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:52 smithi006 bash[26022]: cluster 2023-11-17T06:17:51.008496+0000 mgr.smithi006.odexej (mgr.14176) 2069 : cluster [DBG] pgmap v1395: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:52.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:52 smithi156 bash[29655]: cluster 2023-11-17T06:17:51.008496+0000 mgr.smithi006.odexej (mgr.14176) 2069 : cluster [DBG] pgmap v1395: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:53.644 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:17:53.645 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:17:54.271 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:17:54.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:54 smithi006 bash[26022]: cluster 2023-11-17T06:17:53.009603+0000 mgr.smithi006.odexej (mgr.14176) 2070 : cluster [DBG] pgmap v1396: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:54.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:54 smithi156 bash[29655]: cluster 2023-11-17T06:17:53.009603+0000 mgr.smithi006.odexej (mgr.14176) 2070 : cluster [DBG] pgmap v1396: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:17:55.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:17:55.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:55 smithi006 bash[26022]: audit 2023-11-17T06:17:53.628529+0000 mgr.smithi006.odexej (mgr.14176) 2071 : audit [DBG] from='client.15682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:55.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:55 smithi156 bash[29655]: audit 2023-11-17T06:17:53.628529+0000 mgr.smithi006.odexej (mgr.14176) 2071 : audit [DBG] from='client.15682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:56.406 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:56 smithi006 bash[26022]: cluster 2023-11-17T06:17:55.011047+0000 mgr.smithi006.odexej (mgr.14176) 2072 : cluster [DBG] pgmap v1397: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:56.433 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:56 smithi156 bash[29655]: cluster 2023-11-17T06:17:55.011047+0000 mgr.smithi006.odexej (mgr.14176) 2072 : cluster [DBG] pgmap v1397: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:58.272 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:17:58.272 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:17:58.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:58 smithi006 bash[26022]: cluster 2023-11-17T06:17:57.012121+0000 mgr.smithi006.odexej (mgr.14176) 2073 : cluster [DBG] pgmap v1398: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:58.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:58 smithi156 bash[29655]: cluster 2023-11-17T06:17:57.012121+0000 mgr.smithi006.odexej (mgr.14176) 2073 : cluster [DBG] pgmap v1398: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:17:58.896 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:17:59.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:17:59 smithi006 bash[26022]: audit 2023-11-17T06:17:58.258998+0000 mgr.smithi006.odexej (mgr.14176) 2074 : audit [DBG] from='client.15686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:59.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:17:59 smithi156 bash[29655]: audit 2023-11-17T06:17:58.258998+0000 mgr.smithi006.odexej (mgr.14176) 2074 : audit [DBG] from='client.15686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:17:59.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:00.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:00 smithi006 bash[26022]: cluster 2023-11-17T06:17:59.013290+0000 mgr.smithi006.odexej (mgr.14176) 2075 : cluster [DBG] pgmap v1399: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:00.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:00 smithi156 bash[29655]: cluster 2023-11-17T06:17:59.013290+0000 mgr.smithi006.odexej (mgr.14176) 2075 : cluster [DBG] pgmap v1399: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:02.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:02 smithi006 bash[26022]: cluster 2023-11-17T06:18:01.014691+0000 mgr.smithi006.odexej (mgr.14176) 2076 : cluster [DBG] pgmap v1400: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:02.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:02 smithi156 bash[29655]: cluster 2023-11-17T06:18:01.014691+0000 mgr.smithi006.odexej (mgr.14176) 2076 : cluster [DBG] pgmap v1400: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:02.949 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:02.949 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:03.538 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:04.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:04 smithi156 bash[29655]: audit 2023-11-17T06:18:02.940915+0000 mgr.smithi006.odexej (mgr.14176) 2077 : audit [DBG] from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:04.431 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:04 smithi156 bash[29655]: cluster 2023-11-17T06:18:03.015454+0000 mgr.smithi006.odexej (mgr.14176) 2078 : cluster [DBG] pgmap v1401: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:04.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:04.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:04 smithi006 bash[26022]: audit 2023-11-17T06:18:02.940915+0000 mgr.smithi006.odexej (mgr.14176) 2077 : audit [DBG] from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:04.552 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:04 smithi006 bash[26022]: cluster 2023-11-17T06:18:03.015454+0000 mgr.smithi006.odexej (mgr.14176) 2078 : cluster [DBG] pgmap v1401: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:06.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:06 smithi156 bash[29655]: cluster 2023-11-17T06:18:05.016843+0000 mgr.smithi006.odexej (mgr.14176) 2079 : cluster [DBG] pgmap v1402: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:06.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:06 smithi006 bash[26022]: cluster 2023-11-17T06:18:05.016843+0000 mgr.smithi006.odexej (mgr.14176) 2079 : cluster [DBG] pgmap v1402: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:07.600 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:07.600 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:08.177 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:08.430 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:08 smithi156 bash[29655]: cluster 2023-11-17T06:18:07.017889+0000 mgr.smithi006.odexej (mgr.14176) 2080 : cluster [DBG] pgmap v1403: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:08.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:08 smithi006 bash[26022]: cluster 2023-11-17T06:18:07.017889+0000 mgr.smithi006.odexej (mgr.14176) 2080 : cluster [DBG] pgmap v1403: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:09.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:09.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:09 smithi006 bash[26022]: audit 2023-11-17T06:18:07.591580+0000 mgr.smithi006.odexej (mgr.14176) 2081 : audit [DBG] from='client.15694 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:09.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:09 smithi156 bash[29655]: audit 2023-11-17T06:18:07.591580+0000 mgr.smithi006.odexej (mgr.14176) 2081 : audit [DBG] from='client.15694 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:10.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:10 smithi006 bash[26022]: cluster 2023-11-17T06:18:09.019072+0000 mgr.smithi006.odexej (mgr.14176) 2082 : cluster [DBG] pgmap v1404: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:10.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:10 smithi156 bash[29655]: cluster 2023-11-17T06:18:09.019072+0000 mgr.smithi006.odexej (mgr.14176) 2082 : cluster [DBG] pgmap v1404: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:12.171 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:12.171 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:12.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:12 smithi006 bash[26022]: cluster 2023-11-17T06:18:11.020495+0000 mgr.smithi006.odexej (mgr.14176) 2083 : cluster [DBG] pgmap v1405: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:12.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:12 smithi156 bash[29655]: cluster 2023-11-17T06:18:11.020495+0000 mgr.smithi006.odexej (mgr.14176) 2083 : cluster [DBG] pgmap v1405: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:12.769 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:13.499 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:13 smithi006 bash[26022]: audit 2023-11-17T06:18:12.162839+0000 mgr.smithi006.odexej (mgr.14176) 2084 : audit [DBG] from='client.15698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:13.622 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:13 smithi156 bash[29655]: audit 2023-11-17T06:18:12.162839+0000 mgr.smithi006.odexej (mgr.14176) 2084 : audit [DBG] from='client.15698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:13.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:14.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:14 smithi006 bash[26022]: cluster 2023-11-17T06:18:13.021502+0000 mgr.smithi006.odexej (mgr.14176) 2085 : cluster [DBG] pgmap v1406: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:14.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:14 smithi156 bash[29655]: cluster 2023-11-17T06:18:13.021502+0000 mgr.smithi006.odexej (mgr.14176) 2085 : cluster [DBG] pgmap v1406: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:16.513 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:16 smithi006 bash[26022]: cluster 2023-11-17T06:18:15.022883+0000 mgr.smithi006.odexej (mgr.14176) 2086 : cluster [DBG] pgmap v1407: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:16.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:16 smithi156 bash[29655]: cluster 2023-11-17T06:18:15.022883+0000 mgr.smithi006.odexej (mgr.14176) 2086 : cluster [DBG] pgmap v1407: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:16.953 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:16.953 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:17.573 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:18.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:18.585 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:18 smithi006 bash[26022]: audit 2023-11-17T06:18:16.937258+0000 mgr.smithi006.odexej (mgr.14176) 2087 : audit [DBG] from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:18.585 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:18 smithi006 bash[26022]: cluster 2023-11-17T06:18:17.023763+0000 mgr.smithi006.odexej (mgr.14176) 2088 : cluster [DBG] pgmap v1408: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:18.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:18 smithi156 bash[29655]: audit 2023-11-17T06:18:16.937258+0000 mgr.smithi006.odexej (mgr.14176) 2087 : audit [DBG] from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:18.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:18 smithi156 bash[29655]: cluster 2023-11-17T06:18:17.023763+0000 mgr.smithi006.odexej (mgr.14176) 2088 : cluster [DBG] pgmap v1408: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:20.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:20 smithi006 bash[26022]: cluster 2023-11-17T06:18:19.024987+0000 mgr.smithi006.odexej (mgr.14176) 2089 : cluster [DBG] pgmap v1409: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:20.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:20 smithi156 bash[29655]: cluster 2023-11-17T06:18:19.024987+0000 mgr.smithi006.odexej (mgr.14176) 2089 : cluster [DBG] pgmap v1409: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:21.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:21.896 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:22.530 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:22 smithi006 bash[26022]: cluster 2023-11-17T06:18:21.026441+0000 mgr.smithi006.odexej (mgr.14176) 2090 : cluster [DBG] pgmap v1410: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:22.531 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:22.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:22 smithi156 bash[29655]: cluster 2023-11-17T06:18:21.026441+0000 mgr.smithi006.odexej (mgr.14176) 2090 : cluster [DBG] pgmap v1410: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:23.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:23.543 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:23 smithi006 bash[26022]: audit 2023-11-17T06:18:21.880606+0000 mgr.smithi006.odexej (mgr.14176) 2091 : audit [DBG] from='client.15706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:23.623 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:23 smithi156 bash[29655]: audit 2023-11-17T06:18:21.880606+0000 mgr.smithi006.odexej (mgr.14176) 2091 : audit [DBG] from='client.15706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:24.540 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:24 smithi006 bash[26022]: cluster 2023-11-17T06:18:23.027641+0000 mgr.smithi006.odexej (mgr.14176) 2092 : cluster [DBG] pgmap v1411: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:24.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:24 smithi156 bash[29655]: cluster 2023-11-17T06:18:23.027641+0000 mgr.smithi006.odexej (mgr.14176) 2092 : cluster [DBG] pgmap v1411: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:26.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:26 smithi006 bash[26022]: cluster 2023-11-17T06:18:25.029127+0000 mgr.smithi006.odexej (mgr.14176) 2093 : cluster [DBG] pgmap v1412: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:26.582 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:26.582 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:26.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:26 smithi156 bash[29655]: cluster 2023-11-17T06:18:25.029127+0000 mgr.smithi006.odexej (mgr.14176) 2093 : cluster [DBG] pgmap v1412: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:27.189 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:28.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:28.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:28 smithi006 bash[26022]: audit 2023-11-17T06:18:26.566270+0000 mgr.smithi006.odexej (mgr.14176) 2094 : audit [DBG] from='client.15710 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:28.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:28 smithi006 bash[26022]: cluster 2023-11-17T06:18:27.030171+0000 mgr.smithi006.odexej (mgr.14176) 2095 : cluster [DBG] pgmap v1413: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:28.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:28 smithi156 bash[29655]: audit 2023-11-17T06:18:26.566270+0000 mgr.smithi006.odexej (mgr.14176) 2094 : audit [DBG] from='client.15710 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:28.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:28 smithi156 bash[29655]: cluster 2023-11-17T06:18:27.030171+0000 mgr.smithi006.odexej (mgr.14176) 2095 : cluster [DBG] pgmap v1413: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:30.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:30 smithi006 bash[26022]: cluster 2023-11-17T06:18:29.031374+0000 mgr.smithi006.odexej (mgr.14176) 2096 : cluster [DBG] pgmap v1414: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:30.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:30 smithi156 bash[29655]: cluster 2023-11-17T06:18:29.031374+0000 mgr.smithi006.odexej (mgr.14176) 2096 : cluster [DBG] pgmap v1414: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:31.286 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:31.286 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:31.908 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:32.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:32 smithi006 bash[26022]: cluster 2023-11-17T06:18:31.032257+0000 mgr.smithi006.odexej (mgr.14176) 2097 : cluster [DBG] pgmap v1415: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:32.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:32 smithi006 bash[26022]: audit 2023-11-17T06:18:31.269537+0000 mgr.smithi006.odexej (mgr.14176) 2098 : audit [DBG] from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:32.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:32 smithi156 bash[29655]: cluster 2023-11-17T06:18:31.032257+0000 mgr.smithi006.odexej (mgr.14176) 2097 : cluster [DBG] pgmap v1415: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:32.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:32 smithi156 bash[29655]: audit 2023-11-17T06:18:31.269537+0000 mgr.smithi006.odexej (mgr.14176) 2098 : audit [DBG] from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:32.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:34.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:34 smithi006 bash[26022]: cluster 2023-11-17T06:18:33.033230+0000 mgr.smithi006.odexej (mgr.14176) 2099 : cluster [DBG] pgmap v1416: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:34.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:34 smithi156 bash[29655]: cluster 2023-11-17T06:18:33.033230+0000 mgr.smithi006.odexej (mgr.14176) 2099 : cluster [DBG] pgmap v1416: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:18:35.626 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:35 smithi006 bash[26022]: audit 2023-11-17T06:18:35.002731+0000 mon.smithi006 (mon.0) 1179 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:18:35.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:35 smithi156 bash[29655]: audit 2023-11-17T06:18:35.002731+0000 mon.smithi006 (mon.0) 1179 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:18:36.062 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:36.063 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:16:27.138100Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.wieszl on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: cluster 2023-11-17T06:18:35.034452+0000 mgr.smithi006.odexej (mgr.14176) 2100 : cluster [DBG] pgmap v1417: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: audit 2023-11-17T06:18:35.327853+0000 mon.smithi006 (mon.0) 1180 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: audit 2023-11-17T06:18:35.329490+0000 mon.smithi006 (mon.0) 1181 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: audit 2023-11-17T06:18:35.337337+0000 mon.smithi006 (mon.0) 1182 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: cluster 2023-11-17T06:18:35.338141+0000 mgr.smithi006.odexej (mgr.14176) 2101 : cluster [DBG] pgmap v1418: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: audit 2023-11-17T06:18:35.342886+0000 mon.smithi006 (mon.0) 1183 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:18:36.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: audit 2023-11-17T06:18:35.359765+0000 mon.smithi006 (mon.0) 1184 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:36.416 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:36 smithi006 bash[26022]: cephadm 2023-11-17T06:18:35.363122+0000 mgr.smithi006.odexej (mgr.14176) 2102 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.slgwut on smithi156 2023-11-17T06:18:36.638 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:36.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: cluster 2023-11-17T06:18:35.034452+0000 mgr.smithi006.odexej (mgr.14176) 2100 : cluster [DBG] pgmap v1417: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: audit 2023-11-17T06:18:35.327853+0000 mon.smithi006 (mon.0) 1180 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: audit 2023-11-17T06:18:35.329490+0000 mon.smithi006 (mon.0) 1181 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: audit 2023-11-17T06:18:35.337337+0000 mon.smithi006 (mon.0) 1182 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: cluster 2023-11-17T06:18:35.338141+0000 mgr.smithi006.odexej (mgr.14176) 2101 : cluster [DBG] pgmap v1418: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: audit 2023-11-17T06:18:35.342886+0000 mon.smithi006 (mon.0) 1183 : audit [DBG] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: audit 2023-11-17T06:18:35.359765+0000 mon.smithi006 (mon.0) 1184 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:36.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:36 smithi156 bash[29655]: cephadm 2023-11-17T06:18:35.363122+0000 mgr.smithi006.odexej (mgr.14176) 2102 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi156.slgwut on smithi156 2023-11-17T06:18:37.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:37.652 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:37 smithi006 bash[26022]: audit 2023-11-17T06:18:36.046936+0000 mgr.smithi006.odexej (mgr.14176) 2103 : audit [DBG] from='client.15718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:37.652 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:37 smithi006 bash[26022]: cluster 2023-11-17T06:18:36.334938+0000 mon.smithi006 (mon.0) 1185 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:18:37.652 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:37 smithi006 bash[26022]: cluster 2023-11-17T06:18:36.334986+0000 mon.smithi006 (mon.0) 1186 : cluster [INF] Cluster is now healthy 2023-11-17T06:18:37.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:37 smithi156 bash[29655]: audit 2023-11-17T06:18:36.046936+0000 mgr.smithi006.odexej (mgr.14176) 2103 : audit [DBG] from='client.15718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:37.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:37 smithi156 bash[29655]: cluster 2023-11-17T06:18:36.334938+0000 mon.smithi006 (mon.0) 1185 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:18:37.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:37 smithi156 bash[29655]: cluster 2023-11-17T06:18:36.334986+0000 mon.smithi006 (mon.0) 1186 : cluster [INF] Cluster is now healthy 2023-11-17T06:18:38.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:38 smithi006 bash[26022]: cluster 2023-11-17T06:18:37.338712+0000 mgr.smithi006.odexej (mgr.14176) 2104 : cluster [DBG] pgmap v1419: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:38.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:38 smithi156 bash[29655]: cluster 2023-11-17T06:18:37.338712+0000 mgr.smithi006.odexej (mgr.14176) 2104 : cluster [DBG] pgmap v1419: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:39.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:39 smithi006 bash[26022]: audit 2023-11-17T06:18:39.112658+0000 mon.smithi006 (mon.0) 1187 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.slgwut"}]: dispatch 2023-11-17T06:18:39.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:39 smithi156 bash[29655]: audit 2023-11-17T06:18:39.112658+0000 mon.smithi006 (mon.0) 1187 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi156.slgwut"}]: dispatch 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: cephadm 2023-11-17T06:18:39.111802+0000 mgr.smithi006.odexej (mgr.14176) 2105 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.666 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.667 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: cephadm 2023-11-17T06:18:39.112083+0000 mgr.smithi006.odexej (mgr.14176) 2106 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.668 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: cephadm 2023-11-17T06:18:39.114053+0000 mgr.smithi006.odexej (mgr.14176) 2107 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.slgwut on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.669 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: cephadm 2023-11-17T06:18:39.116601+0000 mgr.smithi006.odexej (mgr.14176) 2108 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.zcqwev on smithi006 2023-11-17T06:18:40.670 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:40 smithi006 bash[26022]: cluster 2023-11-17T06:18:39.339628+0000 mgr.smithi006.odexej (mgr.14176) 2109 : cluster [DBG] pgmap v1420: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: cephadm 2023-11-17T06:18:39.111802+0000 mgr.smithi006.odexej (mgr.14176) 2105 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: cephadm 2023-11-17T06:18:39.112083+0000 mgr.smithi006.odexej (mgr.14176) 2106 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: cephadm 2023-11-17T06:18:39.114053+0000 mgr.smithi006.odexej (mgr.14176) 2107 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi156.slgwut on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi156.slgwut ... 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:40.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:40.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:40.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:40.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: cephadm 2023-11-17T06:18:39.116601+0000 mgr.smithi006.odexej (mgr.14176) 2108 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi006.zcqwev on smithi006 2023-11-17T06:18:40.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:40 smithi156 bash[29655]: cluster 2023-11-17T06:18:39.339628+0000 mgr.smithi006.odexej (mgr.14176) 2109 : cluster [DBG] pgmap v1420: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:40.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:40.860 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:17:31.217720Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.ldxsxc on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-ldxsxc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.ldxsxc\nDeploy daemon haproxy.nfs.foo.smithi156.ldxsxc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.988435Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.ctpzby on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-ctpzby\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.ctpzby\nDeploy daemon haproxy.nfs.foo.smithi006.ctpzby ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:17:34.992853Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.ketzkr on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:18:39.113910Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.slgwut on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\nDeploy daemon haproxy.nfs.foo.smithi156.slgwut ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:41.452 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:42.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:42.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:42 smithi006 bash[26022]: audit 2023-11-17T06:18:40.844248+0000 mgr.smithi006.odexej (mgr.14176) 2110 : audit [DBG] from='client.15722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:42.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:42 smithi006 bash[26022]: cluster 2023-11-17T06:18:41.340212+0000 mgr.smithi006.odexej (mgr.14176) 2111 : cluster [DBG] pgmap v1421: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:42.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:42 smithi156 bash[29655]: audit 2023-11-17T06:18:40.844248+0000 mgr.smithi006.odexej (mgr.14176) 2110 : audit [DBG] from='client.15722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:42.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:42 smithi156 bash[29655]: cluster 2023-11-17T06:18:41.340212+0000 mgr.smithi006.odexej (mgr.14176) 2111 : cluster [DBG] pgmap v1421: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2023-11-17T06:18:43.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:43 smithi006 bash[26022]: audit 2023-11-17T06:18:43.181982+0000 mon.smithi006 (mon.0) 1188 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.zcqwev"}]: dispatch 2023-11-17T06:18:43.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:43 smithi156 bash[29655]: audit 2023-11-17T06:18:43.181982+0000 mon.smithi006 (mon.0) 1188 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi006.zcqwev"}]: dispatch 2023-11-17T06:18:44.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cephadm 2023-11-17T06:18:43.180886+0000 mgr.smithi006.odexej (mgr.14176) 2112 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Traceback (most recent call last): 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: yield (conn, connr) 2023-11-17T06:18:44.682 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: code, '\n'.join(err))) 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.683 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cephadm 2023-11-17T06:18:43.181397+0000 mgr.smithi006.odexej (mgr.14176) 2113 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cephadm 2023-11-17T06:18:43.183193+0000 mgr.smithi006.odexej (mgr.14176) 2114 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.zcqwev on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stdout 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.684 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cephadm 2023-11-17T06:18:43.187202+0000 mgr.smithi006.odexej (mgr.14176) 2115 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.rdyaly on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:18:44.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cephadm 2023-11-17T06:18:43.189729+0000 mgr.smithi006.odexej (mgr.14176) 2116 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.fihxtn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:18:44.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cluster 2023-11-17T06:18:43.190616+0000 mgr.smithi006.odexej (mgr.14176) 2117 : cluster [DBG] pgmap v1422: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:18:44.685 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:44 smithi156 bash[29655]: cluster 2023-11-17T06:18:43.401863+0000 mon.smithi006 (mon.0) 1189 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:18:44.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cephadm 2023-11-17T06:18:43.180886+0000 mgr.smithi006.odexej (mgr.14176) 2112 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.915 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.916 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Traceback (most recent call last): 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: yield (conn, connr) 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: code, '\n'.join(err))) 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.917 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cephadm 2023-11-17T06:18:43.181397+0000 mgr.smithi006.odexej (mgr.14176) 2113 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cephadm 2023-11-17T06:18:43.183193+0000 mgr.smithi006.odexej (mgr.14176) 2114 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi006.zcqwev on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stdout 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: /usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev 2023-11-17T06:18:44.918 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Deploy daemon haproxy.nfs.foo.smithi006.zcqwev ... 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: stat: stderr See 'docker run --help'. 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cephadm 2023-11-17T06:18:43.187202+0000 mgr.smithi006.odexej (mgr.14176) 2115 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi156.rdyaly on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cephadm 2023-11-17T06:18:43.189729+0000 mgr.smithi006.odexej (mgr.14176) 2116 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi006.fihxtn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cluster 2023-11-17T06:18:43.190616+0000 mgr.smithi006.odexej (mgr.14176) 2117 : cluster [DBG] pgmap v1422: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:18:44.919 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:44 smithi006 bash[26022]: cluster 2023-11-17T06:18:43.401863+0000 mon.smithi006 (mon.0) 1189 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:18:45.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:45 smithi006 bash[26022]: cluster 2023-11-17T06:18:45.191472+0000 mgr.smithi006.odexej (mgr.14176) 2118 : cluster [DBG] pgmap v1423: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:18:45.680 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:45 smithi156 bash[29655]: cluster 2023-11-17T06:18:45.191472+0000 mgr.smithi006.odexej (mgr.14176) 2118 : cluster [DBG] pgmap v1423: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2023-11-17T06:18:45.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:45.820 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:18:39.113910Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.slgwut on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\nDeploy daemon haproxy.nfs.foo.smithi156.slgwut ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:18:43.183034Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.zcqwev on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev\nDeploy daemon haproxy.nfs.foo.smithi006.zcqwev ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:18:43.187001Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rdyaly on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:18:43.189591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fihxtn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:46.408 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:46.665 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:46 smithi006 bash[26022]: audit 2023-11-17T06:18:45.803139+0000 mgr.smithi006.odexej (mgr.14176) 2119 : audit [DBG] from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:46.930 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:46 smithi156 bash[29655]: audit 2023-11-17T06:18:45.803139+0000 mgr.smithi006.odexej (mgr.14176) 2119 : audit [DBG] from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:47.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:18:48.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:47 smithi006 bash[26022]: audit 2023-11-17T06:18:46.819539+0000 mon.smithi006 (mon.0) 1190 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:48.165 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:47 smithi006 bash[26022]: cluster 2023-11-17T06:18:47.192500+0000 mgr.smithi006.odexej (mgr.14176) 2120 : cluster [DBG] pgmap v1424: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:48.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:47 smithi156 bash[29655]: audit 2023-11-17T06:18:46.819539+0000 mon.smithi006 (mon.0) 1190 : audit [INF] from='mgr.14176 172.21.15.6:0/3273255719' entity='mgr.smithi006.odexej' 2023-11-17T06:18:48.180 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:47 smithi156 bash[29655]: cluster 2023-11-17T06:18:47.192500+0000 mgr.smithi006.odexej (mgr.14176) 2120 : cluster [DBG] pgmap v1424: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:50.415 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:50 smithi006 bash[26022]: cluster 2023-11-17T06:18:49.193841+0000 mgr.smithi006.odexej (mgr.14176) 2121 : cluster [DBG] pgmap v1425: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:50.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-11-17T06:18:50.503 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T05:32:28.280394Z", "last_refresh": "2023-11-17T06:12:53.726074Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T05:32:20.605834Z", "last_refresh": "2023-11-17T06:12:52.295941Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T05:32:24.432488Z", "last_refresh": "2023-11-17T06:12:53.726491Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:17:34.996922Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.hpzcvh on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:18:39.113910Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi156.slgwut on smithi156: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi156-slgwut\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi156.slgwut\nDeploy daemon haproxy.nfs.foo.smithi156.slgwut ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi156 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:18:43.183034Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi006.zcqwev on smithi006: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy-nfs-foo-smithi006-zcqwev\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-haproxy.nfs.foo.smithi006.zcqwev\nDeploy daemon haproxy.nfs.foo.smithi006.zcqwev ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi006 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:18:43.187001Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi156.rdyaly on smithi156: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:18:43.189591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi006.fihxtn on smithi006: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2999, "monitor_port": 9999, "virtual_ip": "10.0.31.6/16"}, "status": {"created": "2023-11-17T05:41:00.659423Z", "ports": [2999, 9999], "running": 0, "size": 4, "virtual_ip": "10.0.31.6/16"}}, {"events": ["2023-11-17T05:40:55.183979Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T05:40:55.178208Z", "last_refresh": "2023-11-17T06:12:52.297318Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T05:32:18.690705Z", "last_refresh": "2023-11-17T06:12:52.296196Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:19.888948Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi006:172.21.15.6=smithi006", "smithi156:172.21.15.156=smithi156"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T05:34:19.882929Z", "last_refresh": "2023-11-17T06:12:52.296364Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T05:41:00.657974Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12999}, "status": {"created": "2023-11-17T05:41:00.654175Z", "last_refresh": "2023-11-17T06:12:53.728173Z", "ports": [12999], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T05:32:26.276503Z", "last_refresh": "2023-11-17T06:12:52.296524Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T05:34:51.007864Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T05:34:51.000616Z", "last_refresh": "2023-11-17T06:12:52.296681Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T05:32:22.472177Z", "last_refresh": "2023-11-17T06:12:53.726917Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:18:50.681 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:50 smithi156 bash[29655]: cluster 2023-11-17T06:18:49.193841+0000 mgr.smithi006.odexej (mgr.14176) 2121 : cluster [DBG] pgmap v1425: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:51.100 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:18:51.101 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1119, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-11-17T06:18:52.875 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b462bf9405104393bd926a4018142ca5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1119, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-11-17T06:18:52.878 DEBUG:teuthology.run_tasks:Unwinding manager vip 2023-11-17T06:18:52.890 INFO:tasks.vip:Removing 10.0.15.6 (and any VIPs) on smithi006.front.sepia.ceph.com iface ens1f0... 2023-11-17T06:18:52.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr del 10.0.15.6/16 dev ens1f0 2023-11-17T06:18:52.907 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:52 smithi156 bash[29655]: audit 2023-11-17T06:18:50.486596+0000 mgr.smithi006.odexej (mgr.14176) 2122 : audit [DBG] from='client.15730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:52.907 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:52 smithi156 bash[29655]: cluster 2023-11-17T06:18:51.195263+0000 mgr.smithi006.odexej (mgr.14176) 2123 : cluster [DBG] pgmap v1426: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:52.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:52 smithi006 bash[26022]: audit 2023-11-17T06:18:50.486596+0000 mgr.smithi006.odexej (mgr.14176) 2122 : audit [DBG] from='client.15730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:18:52.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:52 smithi006 bash[26022]: cluster 2023-11-17T06:18:51.195263+0000 mgr.smithi006.odexej (mgr.14176) 2123 : cluster [DBG] pgmap v1426: 97 pgs: 97 active+clean; 4.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2023-11-17T06:18:52.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr del 10.0.31.6/16 dev ens1f0 2023-11-17T06:18:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-17T06:18:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:RTNETLINK answers: Cannot assign requested address 2023-11-17T06:18:52.992 INFO:tasks.vip:Removing 10.0.15.156 (and any VIPs) on smithi156.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T06:18:52.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr del 10.0.15.156/16 dev enp3s0f1 2023-11-17T06:18:53.013 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr del 10.0.31.6/16 dev enp3s0f1 2023-11-17T06:18:53.069 INFO:teuthology.orchestra.run.smithi156.stderr:RTNETLINK answers: Cannot assign requested address 2023-11-17T06:18:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-17T06:18:53.070 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-17T06:18:53.086 INFO:tasks.cephadm:Teardown begin 2023-11-17T06:18:53.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:18:53.105 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:18:53.128 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-17T06:18:53.128 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T06:18:53.151 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T06:18:53.179 INFO:tasks.cephadm:Stopping all daemons... 2023-11-17T06:18:53.179 INFO:tasks.cephadm.mon.smithi006:Stopping mon.smithi006... 2023-11-17T06:18:53.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006 2023-11-17T06:18:53.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:53 smithi006 systemd[1]: Stopping Ceph mon.smithi006 for 7d14d910-850a-11ee-8dbe-212e2dc638e7... 2023-11-17T06:18:53.501 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:53 smithi006 bash[122997]: Error response from daemon: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-mon.smithi006 2023-11-17T06:18:53.501 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:53 smithi006 bash[26022]: debug 2023-11-17T06:18:53.394+0000 7fe985b0f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi006 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T06:18:53.501 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Nov 17 06:18:53 smithi006 bash[26022]: debug 2023-11-17T06:18:53.394+0000 7fe985b0f700 -1 mon.smithi006@0(leader) e2 *** Got Signal Terminated *** 2023-11-17T06:18:54.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi006.service' 2023-11-17T06:18:54.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T06:18:54.536 INFO:tasks.cephadm.mon.smithi006:Stopped mon.smithi006 2023-11-17T06:18:54.536 INFO:tasks.cephadm.mon.smithi156:Stopping mon.smithi156... 2023-11-17T06:18:54.536 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi156 2023-11-17T06:18:54.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:54 smithi156 systemd[1]: Stopping Ceph mon.smithi156 for 7d14d910-850a-11ee-8dbe-212e2dc638e7... 2023-11-17T06:18:54.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:54 smithi156 bash[23845]: Error response from daemon: No such container: ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7-mon.smithi156 2023-11-17T06:18:54.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:54 smithi156 bash[29655]: debug 2023-11-17T06:18:54.736+0000 7fe61a637700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi156 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T06:18:54.915 INFO:journalctl@ceph.mon.smithi156.smithi156.stdout:Nov 17 06:18:54 smithi156 bash[29655]: debug 2023-11-17T06:18:54.736+0000 7fe61a637700 -1 mon.smithi156@1(peon) e2 *** Got Signal Terminated *** 2023-11-17T06:18:55.567 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7d14d910-850a-11ee-8dbe-212e2dc638e7@mon.smithi156.service' 2023-11-17T06:18:55.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T06:18:55.607 INFO:tasks.cephadm.mon.smithi156:Stopped mon.smithi156 2023-11-17T06:18:55.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-17T06:19:54.561 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-17T06:21:09.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:21:09.719 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:21:09.729 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-17T06:21:09.732 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/ubuntu@smithi006.front.sepia.ceph.com/crash 2023-11-17T06:21:09.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T06:21:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:tar: /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T06:21:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:tar: Error is not recoverable: exiting now 2023-11-17T06:21:09.777 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/ubuntu@smithi156.front.sepia.ceph.com/crash 2023-11-17T06:21:09.778 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T06:21:09.790 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T06:21:09.791 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2023-11-17T06:21:09.792 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-17T06:21:09.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-11-17T06:21:09.833 INFO:teuthology.orchestra.run.smithi006.stderr:grep: /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-17T06:21:09.835 INFO:tasks.cephadm:Compressing logs... 2023-11-17T06:21:09.836 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T06:21:09.880 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T06:21:09.892 INFO:teuthology.orchestra.run.smithi006.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-11-17T06:21:09.893 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T06:21:09.894 INFO:teuthology.orchestra.run.smithi156.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-11-17T06:21:09.894 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-17T06:21:09.894 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi006.rwbcnj.log 2023-11-17T06:21:09.894 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi006.log 2023-11-17T06:21:09.895 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T06:21:09.896 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi006.rwbcnj.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.7.log 2023-11-17T06:21:09.896 INFO:teuthology.orchestra.run.smithi006.stderr: 60.3% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi006.rwbcnj.log.gz 2023-11-17T06:21:09.896 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi006.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi006.odexej.log 2023-11-17T06:21:09.898 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.1.log 2023-11-17T06:21:09.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.7.log: /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi006.odexej.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.5.log 2023-11-17T06:21:09.899 INFO:teuthology.orchestra.run.smithi006.stderr:gzip /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.1.log: -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.3.log /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.5.log: 2023-11-17T06:21:09.899 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.6.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi156.fphsnp.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi156.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi156.fphsnp.log: -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi156.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.0.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.2.log 2023-11-17T06:21:09.900 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi156.gmiiei.log 2023-11-17T06:21:09.901 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.4.log 2023-11-17T06:21:09.901 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.2.log: /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi156.gmiiei.log: 84.5% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mds.foofs.smithi156.gmiiei.log.gz 2023-11-17T06:21:09.907 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.3.log: 92.5% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-17T06:21:09.908 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.4.log: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T06:21:09.909 INFO:teuthology.orchestra.run.smithi156.stderr: 92.4% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-17T06:21:09.914 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi156.fphsnp.log.gz 2023-11-17T06:21:09.920 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T06:21:10.459 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi006.odexej.log.gz 2023-11-17T06:21:10.504 INFO:teuthology.orchestra.run.smithi156.stderr: 92.7% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi156.log.gz 2023-11-17T06:21:10.543 INFO:teuthology.orchestra.run.smithi156.stderr: 93.1% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.2.log.gz 2023-11-17T06:21:10.583 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.5.log.gz 2023-11-17T06:21:10.651 INFO:teuthology.orchestra.run.smithi006.stderr: 93.0% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.7.log.gz 2023-11-17T06:21:10.719 INFO:teuthology.orchestra.run.smithi156.stderr: 92.9% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.6.log.gz 2023-11-17T06:21:10.733 INFO:teuthology.orchestra.run.smithi156.stderr: 93.0% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.0.log.gz 2023-11-17T06:21:10.762 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.1.log.gz 2023-11-17T06:21:10.777 INFO:teuthology.orchestra.run.smithi156.stderr: 92.9% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.4.log.gz 2023-11-17T06:21:10.777 INFO:teuthology.orchestra.run.smithi156.stderr: 2023-11-17T06:21:10.778 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.893s 2023-11-17T06:21:10.778 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m3.678s 2023-11-17T06:21:10.778 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.175s 2023-11-17T06:21:11.027 INFO:teuthology.orchestra.run.smithi006.stderr: 93.4% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-osd.3.log.gz 2023-11-17T06:21:11.316 INFO:teuthology.orchestra.run.smithi006.stderr: 89.8% -- replaced with /var/log/ceph/7d14d910-850a-11ee-8dbe-212e2dc638e7/ceph-mon.smithi006.log.gz 2023-11-17T06:21:11.317 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-11-17T06:21:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m1.435s 2023-11-17T06:21:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m5.277s 2023-11-17T06:21:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.205s 2023-11-17T06:21:11.318 INFO:tasks.cephadm:Archiving logs... 2023-11-17T06:21:11.319 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/ubuntu@smithi006.front.sepia.ceph.com/log 2023-11-17T06:21:11.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T06:21:12.112 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/ubuntu@smithi156.front.sepia.ceph.com/log 2023-11-17T06:21:12.113 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T06:21:12.659 INFO:tasks.cephadm:Removing cluster... 2023-11-17T06:21:12.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 --force 2023-11-17T06:21:14.064 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7d14d910-850a-11ee-8dbe-212e2dc638e7 --force 2023-11-17T06:21:15.448 INFO:tasks.cephadm:Removing cephadm ... 2023-11-17T06:21:15.448 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T06:21:15.455 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T06:21:15.462 INFO:tasks.cephadm:Teardown complete 2023-11-17T06:21:15.462 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2023-11-17T06:21:15.474 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_1... 2023-11-17T06:21:15.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_1 2023-11-17T06:21:15.778 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-11-17T06:21:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:15.779 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_2... 2023-11-17T06:21:15.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_2 2023-11-17T06:21:16.066 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-11-17T06:21:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:16.067 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_3... 2023-11-17T06:21:16.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_3 2023-11-17T06:21:16.346 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-11-17T06:21:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:16.347 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_4... 2023-11-17T06:21:16.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_4 2023-11-17T06:21:16.666 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-11-17T06:21:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:16.668 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-11-17T06:21:16.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/scratch_devs 2023-11-17T06:21:16.683 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi156:/dev/vg_nvme/lv_1... 2023-11-17T06:21:16.684 DEBUG:teuthology.orchestra.run.smithi156:> sudo nvme disconnect -n lv_1 2023-11-17T06:21:16.986 INFO:teuthology.orchestra.run.smithi156.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-11-17T06:21:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:16.987 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi156:/dev/vg_nvme/lv_2... 2023-11-17T06:21:16.988 DEBUG:teuthology.orchestra.run.smithi156:> sudo nvme disconnect -n lv_2 2023-11-17T06:21:17.309 INFO:teuthology.orchestra.run.smithi156.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-11-17T06:21:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:17.311 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi156:/dev/vg_nvme/lv_3... 2023-11-17T06:21:17.312 DEBUG:teuthology.orchestra.run.smithi156:> sudo nvme disconnect -n lv_3 2023-11-17T06:21:17.629 INFO:teuthology.orchestra.run.smithi156.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-11-17T06:21:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:17.631 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi156:/dev/vg_nvme/lv_4... 2023-11-17T06:21:17.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo nvme disconnect -n lv_4 2023-11-17T06:21:17.945 INFO:teuthology.orchestra.run.smithi156.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-11-17T06:21:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:17.947 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-17T06:21:17.948 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/scratch_devs 2023-11-17T06:21:17.963 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-17T06:21:17.973 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-17T06:21:17.974 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T06:21:17.976 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T06:21:17.996 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T06:21:17.996 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2023-11-17T06:21:17.996 INFO:teuthology.orchestra.run.smithi006.stdout:*hv01.front.sepi 44.4.53.2 2 u 8 64 377 0.079 1.975 0.616 2023-11-17T06:21:17.996 INFO:teuthology.orchestra.run.smithi006.stdout:+hv02.front.sepi 63.231.80.2 3 u 15 64 377 0.117 3.821 0.767 2023-11-17T06:21:17.996 INFO:teuthology.orchestra.run.smithi006.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.098 2.403 0.586 2023-11-17T06:21:17.997 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-17T06:21:18.032 INFO:teuthology.orchestra.run.smithi156.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T06:21:18.032 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================== 2023-11-17T06:21:18.033 INFO:teuthology.orchestra.run.smithi156.stdout:*hv01.front.sepi 44.4.53.2 2 u 6 64 377 0.089 -0.337 0.196 2023-11-17T06:21:18.033 INFO:teuthology.orchestra.run.smithi156.stdout:+hv02.front.sepi 63.231.80.2 3 u 67 64 377 0.085 1.227 0.187 2023-11-17T06:21:18.033 INFO:teuthology.orchestra.run.smithi156.stdout:+hv03.front.sepi 74.6.168.72 3 u 67 64 377 0.085 0.829 0.122 2023-11-17T06:21:18.033 INFO:teuthology.orchestra.run.smithi156.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-17T06:21:18.034 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-17T06:21:18.048 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-17T06:21:18.049 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-17T06:21:18.126 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-17T06:21:18.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-17T06:21:18.206 INFO:teuthology.task.internal:Duration was 3395.946878 seconds 2023-11-17T06:21:18.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-17T06:21:18.252 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-17T06:21:18.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T06:21:18.255 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T06:21:18.305 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-17T06:21:18.305 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2023-11-17T06:21:18.305 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T06:21:18.316 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-11-17T06:21:18.317 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T06:21:18.362 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-17T06:21:18.362 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T06:21:18.365 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T06:21:18.460 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-17T06:21:18.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T06:21:18.463 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T06:21:18.624 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-17T06:21:18.645 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-17T06:21:18.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T06:21:18.659 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T06:21:18.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-17T06:21:18.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T06:21:18.703 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T06:21:18.715 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2023-11-17T06:21:18.731 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-11-17T06:21:18.754 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T06:21:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:18.795 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T06:21:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:21:18.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-17T06:21:18.815 INFO:teuthology.task.internal:Transferring archived files... 2023-11-17T06:21:18.817 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/smithi006 2023-11-17T06:21:18.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T06:21:18.873 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284/remote/smithi156 2023-11-17T06:21:18.874 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T06:21:18.899 INFO:teuthology.task.internal:Removing archive directory... 2023-11-17T06:21:18.899 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T06:21:18.907 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T06:21:18.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-17T06:21:18.955 INFO:teuthology.task.internal:Not uploading archives. 2023-11-17T06:21:18.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-17T06:21:19.004 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-17T06:21:19.004 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T06:21:19.007 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T06:21:19.012 INFO:teuthology.orchestra.run.smithi006.stdout: 659549 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 06:21 /home/ubuntu/cephtest 2023-11-17T06:21:19.015 INFO:teuthology.orchestra.run.smithi156.stdout: 659549 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 06:21 /home/ubuntu/cephtest 2023-11-17T06:21:19.017 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-17T06:21:19.040 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-17T06:21:19.140 INFO:teuthology.nuke:Checking targets against current locks 2023-11-17T06:21:19.171 DEBUG:teuthology.nuke:shortname: smithi006 2023-11-17T06:21:19.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T06:21:19.198 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 05:14:41.215224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2023-11-17T06:21:19.250 INFO:teuthology.orchestra.console.smithi006:Power off 2023-11-17T06:21:19.250 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T06:21:19.270 DEBUG:teuthology.nuke:shortname: smithi156 2023-11-17T06:21:19.271 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T06:21:19.288 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2023-11-17T06:21:19.288 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:19.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 05:14:41.217500', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2023-11-17T06:21:19.322 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-11-17T06:21:19.362 INFO:teuthology.orchestra.console.smithi156:Power off 2023-11-17T06:21:19.363 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T06:21:19.393 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2023-11-17T06:21:19.393 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:19.430 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-17T06:21:23.324 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:23.431 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:23.442 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-11-17T06:21:23.542 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-17T06:21:27.446 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:27.543 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:27.558 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-11-17T06:21:27.657 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2023-11-17T06:21:27.757 INFO:teuthology.orchestra.console.smithi156:Power off completed 2023-11-17T06:21:31.562 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T06:21:31.673 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-11-17T06:21:31.774 INFO:teuthology.orchestra.console.smithi006:Power off completed 2023-11-17T06:21:31.878 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} duration: 3395.946877717972 failure_reason: reached maximum tries (301) after waiting for 300 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b462bf9405104393bd926a4018142ca5 status: fail success: false 2023-11-17T06:21:31.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T06:21:31.960 INFO:teuthology.run:FAIL